diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 00000000..6e2ef54c --- /dev/null +++ b/.editorconfig @@ -0,0 +1,19 @@ +# EditorConfig docs: https://editorconfig.org/ +root = true + +# Default rules for all files. +[*] +charset = utf-8 +end_of_line = lf +insert_final_newline = true +indent_style = space +indent_size = 2 +trim_trailing_whitespace = true + +[*.md] +# keep intentional trailing spaces for hard line breaks. +trim_trailing_whitespace = false + +# YAML: 2-space indentation. +[*.{yml,yaml}] +indent_size = 2 diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 016f12c4..6ead3857 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -7,5 +7,5 @@ # review when someone opens a pull request. * @tomcollins @BillyRuffian @chrilliams -# All content +# All content /content/ @tomcollins @BillyRuffian @chrilliams diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index ae82bff1..947b29ee 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,9 +1,7 @@ ## Content Guidelines -Post content must not include: - -- identifiers for production infrastructure e.g. IP addresses, network ranges, cloud provider IDs etc -- data relating to real customers or staff members +Post content must have a strong technical focus with other engineers being the primary audience. +There are other communication channels for non-technical content and delivery or service updates such as [dvladigital.blog.gov.uk](https://dvladigital.blog.gov.uk/). ### Content Guidelines - Security @@ -11,18 +9,39 @@ Consider if the content of your post, including and images, has any security con - would the information be revealing to an attacker? - is the content too revealing of backend processes? -- do we need to make make specific details public? +- do we need to make specific details public? + +Post content must not include: + +- identifiers for production infrastructure e.g. IP addresses, network ranges, cloud provider IDs etc +- data relating to real customers or staff members + +If you have any doubt please seek support from the cybersecurity team before publishing your content. + +### Content Guidelines - Communication + +You must consider if the content of your post could cause damage to the reputation of the organisation. + +Your post content should not describe, or be directly related to: + +- changes that impact staff experience e.g. automation of work, changes to the types of work undertaken +- changes that impact customers experience e.g. changes to customer process, customer waiting times +- specific DVLA internal or external services -If you have any doubt please seek support from the cyber security team before publishing your content. +If you have any doubt please seek support from the [External Communications](mailto:external.comms@dvla.gov.uk) team before publishing your content. -### Content Guidelines - Reputation +### Content Guidelines - Accessibility -You must consider if the content of your post could cause damage to the reputation of the organisation. +All content must be accessible to all users. This includes: -If you have any doubt please seek support from the communications team before publishing your content. +- using appropriate headings to structure content +- using descriptive alt text or captions for images and figures +- using descriptive link text for hyperlinks +- using appropriate contrast for text and images +- using helpful formatting such as lists and tables to structure content -## Approval Process +While not DVLA-specific, the GOV.UK Content and Publishing Service has some useful [guidance on formatting content](https://guidance.publishing.service.gov.uk/formatting-content/) to [be accessible and clear](https://guidance.publishing.service.gov.uk/writing-to-gov-uk-standards/tone-of-voice/clear-structure/). -All changes to content must be approved by at least one person from the engineering review control group. This takes the form of a github pull request and is automatically enforced. +Given our audience is primarily other engineers, consider the use of technical language and jargon. -Formal approval within the wider organisation is not required by default but may be requested based on e.g. security or reputational concerns. +Aim to use clear and concise language that is accessible to a wide range of readers, including those who may not be familiar with specific technologies or concepts. diff --git a/.github/workflows/gh-pages.yml b/.github/workflows/gh-pages.yml index a1d731d6..0ab4970c 100644 --- a/.github/workflows/gh-pages.yml +++ b/.github/workflows/gh-pages.yml @@ -41,4 +41,3 @@ jobs: publish_dir: ./public allow_empty_commit: true # force_orphan: true - diff --git a/.gitignore b/.gitignore index cbccc036..d8821064 100644 --- a/.gitignore +++ b/.gitignore @@ -3,4 +3,12 @@ node_modules .DS_Store **/*/.DS_Store public -.idea \ No newline at end of file +.idea/* +!.idea/externalDependencies.xml +!.idea/prettier.xml +!.idea/codeStyles/ +!.idea/codeStyles/codeStyleConfig.xml +!.idea/codeStyles/Project.xml +!.idea/dictionaries/ +!.idea/dictionaries/project.xml +resources diff --git a/.idea/codeStyles/Project.xml b/.idea/codeStyles/Project.xml new file mode 100644 index 00000000..8adbf206 --- /dev/null +++ b/.idea/codeStyles/Project.xml @@ -0,0 +1,15 @@ + + + + + + + + diff --git a/.idea/codeStyles/codeStyleConfig.xml b/.idea/codeStyles/codeStyleConfig.xml new file mode 100644 index 00000000..0f7bc519 --- /dev/null +++ b/.idea/codeStyles/codeStyleConfig.xml @@ -0,0 +1,5 @@ + + + + diff --git a/.idea/dictionaries/project.xml b/.idea/dictionaries/project.xml new file mode 100644 index 00000000..80f2b439 --- /dev/null +++ b/.idea/dictionaries/project.xml @@ -0,0 +1,16 @@ + + + + DSIT + DVLA's + dvladigital + fullpage + gsub + kaping + mailsac + markdownlint + milvus + ractors + + + diff --git a/.idea/externalDependencies.xml b/.idea/externalDependencies.xml new file mode 100644 index 00000000..48fe9c37 --- /dev/null +++ b/.idea/externalDependencies.xml @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/.idea/prettier.xml b/.idea/prettier.xml new file mode 100644 index 00000000..7584838b --- /dev/null +++ b/.idea/prettier.xml @@ -0,0 +1,8 @@ + + + + + diff --git a/.markdownlint.yaml b/.markdownlint.yaml new file mode 100644 index 00000000..7dd5921c --- /dev/null +++ b/.markdownlint.yaml @@ -0,0 +1,18 @@ +# Docs: https://github.com/DavidAnson/markdownlint/blob/main/README.md#configuration +default: true + +# MD033/no-inline-html +# Rule: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md033---no-inline-html +# Hugo shortcodes (e.g. {{< figure-a11y ... >}}) are treated as inline HTML by markdownlint. +MD033: false + +# MD013/line-length +# Rule: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md013---line-length +# Avoid erroring on long URLs, shortcode lines, and code examples. +MD013: false + +# MD041/first-line-heading/first-line-h1 +# Rule: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md041---first-line-headingfirst-line-h1 +# Front matter + optional title conventions in posts and pages mean that the first line of the markdown +# file isn't always a heading, so disable this rule. +MD041: false diff --git a/.prettierrc.json b/.prettierrc.json new file mode 100644 index 00000000..24455ee9 --- /dev/null +++ b/.prettierrc.json @@ -0,0 +1,10 @@ +{ + "$schema": "https://json.schemastore.org/prettierrc", + "_docs": "Prettier configuration docs: https://prettier.io/docs/configuration", + "printWidth": 120, + "proseWrap": "preserve", + "tabWidth": 2, + "useTabs": false, + "singleQuote": false, + "trailingComma": "none" +} diff --git a/.run/Hugo_Build.run.xml b/.run/Hugo_Build.run.xml new file mode 100644 index 00000000..313ce98c --- /dev/null +++ b/.run/Hugo_Build.run.xml @@ -0,0 +1,13 @@ + + + + diff --git a/.run/Hugo_Serve.run.xml b/.run/Hugo_Serve.run.xml new file mode 100644 index 00000000..ed292e45 --- /dev/null +++ b/.run/Hugo_Serve.run.xml @@ -0,0 +1,13 @@ + + + + diff --git a/.vscode/extensions.json b/.vscode/extensions.json new file mode 100644 index 00000000..495b1f23 --- /dev/null +++ b/.vscode/extensions.json @@ -0,0 +1,8 @@ +{ + "recommendations": [ + "budparr.language-hugo-vscode", + "editorconfig.editorconfig", + "davidanson.vscode-markdownlint", + "esbenp.prettier-vscode" + ] +} diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 00000000..a0c9c802 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,33 @@ +{ + "editor.formatOnSave": true, + "editor.detectIndentation": false, + + "prettier.requireConfig": true, + "prettier.useEditorConfig": true, + "prettier.configPath": ".prettierrc.json", + + "markdownlint.configFile": ".markdownlint.yaml", + "markdownlint.run": "onSave", + + "[markdown]": { + "editor.defaultFormatter": "esbenp.prettier-vscode", + "editor.formatOnSave": true, + "editor.quickSuggestions": { + "comments": "on", + "strings": "on", + "other": "on" + } + }, + + "[json]": { + "editor.defaultFormatter": "esbenp.prettier-vscode" + }, + + "[jsonc]": { + "editor.defaultFormatter": "esbenp.prettier-vscode" + }, + + "[yaml]": { + "editor.defaultFormatter": "esbenp.prettier-vscode" + } +} diff --git a/.vscode/tasks.json b/.vscode/tasks.json new file mode 100644 index 00000000..51bc32b4 --- /dev/null +++ b/.vscode/tasks.json @@ -0,0 +1,28 @@ +{ + "version": "2.0.0", + "tasks": [ + { + "label": "Hugo: Build", + "type": "shell", + "command": "hugo", + "options": { + "cwd": "${workspaceFolder}" + }, + "group": { + "kind": "build", + "isDefault": true + }, + "problemMatcher": [] + }, + { + "label": "Hugo: Serve", + "type": "shell", + "command": "hugo server -D", + "options": { + "cwd": "${workspaceFolder}" + }, + "isBackground": true, + "problemMatcher": [] + } + ] +} diff --git a/README.md b/README.md index 9ece24cb..38f3c584 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,8 @@ ## Content Guidelines -Post content must have a strong technical focus with other engineers being the primary audience. There are other communication channels for non-technical content and delivery or service updates such as [dvladigital.blog.gov.uk](https://dvladigital.blog.gov.uk/). +Post content must have a strong technical focus with other engineers being the primary audience. +There are other communication channels for non-technical content and delivery or service updates such as [dvladigital.blog.gov.uk](https://dvladigital.blog.gov.uk/). ### Content Guidelines - Security @@ -12,14 +13,14 @@ Consider if the content of your post, including and images, has any security con - would the information be revealing to an attacker? - is the content too revealing of backend processes? -- do we need to make make specific details public? +- do we need to make specific details public? Post content must not include: - identifiers for production infrastructure e.g. IP addresses, network ranges, cloud provider IDs etc - data relating to real customers or staff members -If you have any doubt please seek support from the cyber security team before publishing your content. +If you have any doubt please seek support from the cybersecurity team before publishing your content. ### Content Guidelines - Communication @@ -33,6 +34,22 @@ Your post content should not describe, or be directly related to: If you have any doubt please seek support from the [External Communications](mailto:external.comms@dvla.gov.uk) team before publishing your content. +### Content Guidelines - Accessibility + +All content must be accessible to all users. This includes: + +- using appropriate headings to structure content +- using descriptive alt text or captions for images and figures +- using descriptive link text for hyperlinks +- using appropriate contrast for text and images +- using helpful formatting such as lists and tables to structure content + +While not DVLA-specific, the GOV.UK Content and Publishing Service has some useful [guidance on formatting content](https://guidance.publishing.service.gov.uk/formatting-content/) to [be accessible and clear](https://guidance.publishing.service.gov.uk/writing-to-gov-uk-standards/tone-of-voice/clear-structure/). + +Given our audience is primarily other engineers, consider the use of technical language and jargon. + +Aim to use clear and concise language that is accessible to a wide range of readers, including those who may not be familiar with specific technologies or concepts. + ## Approval Process All changes to content must be approved by at least one person from the engineering review control group. This takes the form of a github pull request and is automatically enforced. @@ -72,3 +89,13 @@ git checkout master 'ensure you are on the master branch' git pull 'get latest changes from master branch' hugo server -D 'starts site on localhost:1313' ``` + +## Using Code Quality Tools + +This repo has been set up with the following developer tools to ensure code quality and consistency, with defaults set up for IntelliJ and VSCode users who have the relevant plugins installed. + +- [EditorConfig](https://editorconfig.org/) for consistent IDE configurations across different editors and IDEs (line endings, indentation, charset etc) +- [Prettier](https://prettier.io/) for applying formatting rules automatically (e.g. line length, spacing etc) +- [Markdownlint](https://github.com/DavidAnson/markdownlint) for applying markdown style rules that prettier can't pick up + +Tool or IDE usage is not enforced, but it is recommended to use these tools where possible to ensure consistency across the codebase and to avoid formatting inconsistencies when submitting pull requests. diff --git a/archetypes/default.md b/archetypes/default.md index 00e77bd7..26f317f3 100644 --- a/archetypes/default.md +++ b/archetypes/default.md @@ -3,4 +3,3 @@ title: "{{ replace .Name "-" " " | title }}" date: {{ .Date }} draft: true --- - diff --git a/assets/css/extended/media.css b/assets/css/extended/media.css new file mode 100644 index 00000000..e529232c --- /dev/null +++ b/assets/css/extended/media.css @@ -0,0 +1,17 @@ +.post-content img { + /* Maintain aspect ratio of images */ + width: 100%; + height: auto; +} + +.post-content figure.align-center img { + /* Avoid stretching centered images */ + margin-left: auto; + margin-right: auto; + width: auto; +} + +.post-content figure audio { + /* Give the audio player some room to breathe */ + margin: 1.5rem 0 1rem; +} diff --git a/assets/css/extended/page.css b/assets/css/extended/page.css new file mode 100644 index 00000000..ca5a3906 --- /dev/null +++ b/assets/css/extended/page.css @@ -0,0 +1,9 @@ +a:hover { + /* Make sure links are underlined on hover to improve accessibility, focus styles are already present */ + text-decoration: underline; +} + +.first-entry { + /* Remove the chin off the first entry on the home page */ + min-height: auto; +} diff --git a/config.yml b/config.yml index 1f411bf8..67ce3d3e 100644 --- a/config.yml +++ b/config.yml @@ -15,6 +15,9 @@ googleAnalytics: G-WH4N9FEEE8 minify: disableXML: true minifyOutput: true + tdewolff: + html: + keepQuotes: true params: env: production @@ -88,10 +91,10 @@ params: # analytics: # google: # SiteVerificationTag: "XYZabc" - # bing: - # SiteVerificationTag: "XYZabc" - # yandex: - # SiteVerificationTag: "XYZabc" + # bing: + # SiteVerificationTag: "XYZabc" + # yandex: + # SiteVerificationTag: "XYZabc" cover: hidden: true # hide everywhere but not in structured data @@ -112,7 +115,7 @@ params: distance: 1000 threshold: 0.4 minMatchCharLength: 0 - keys: ["title", "permalink", "summary", "content"] + keys: ["title", "description", "permalink", "summary", "content"] menu: main: - identifier: posts diff --git a/content/open-source/index.md b/content/open-source/index.md index f64fdde1..ebc1e3ee 100644 --- a/content/open-source/index.md +++ b/content/open-source/index.md @@ -4,7 +4,7 @@ date: 2023-02-27T18:03:22Z draft: false --- -Ok so we don't code entirely [in the open](https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open) at the DVLA we do like to share useful libraries and utilities when we can. You can find a list of recent open-source projects below. +OK so we don't code entirely [in the open](https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open) at the DVLA we do like to share useful libraries and utilities when we can. You can find a list of recent open-source projects below. ## AWS @@ -58,13 +58,13 @@ This gem has pre-configured browser drivers that you can use out-of-the-box for Ka-Ping! An Idiomatic ruby way to construct ElasticSearch queries. -# Dynamics 365 +## Dynamics 365 ### [dataverse-helper](https://github.com/dvla/dataverse-helper) This gem helps you integrate with Microsoft Dynamics using Microsoft Dataverse Web API. You can create, retrieve, delete or update a record without worrying about authentications as it's automatically managed behind the scenes. -# JavaScript +## JavaScript ### [postman-paf-js](https://github.com/dvla/postman-paf-js) diff --git a/content/posts/2022-10-working-with-json-schema/images/code-gen.jpg b/content/posts/2022-10-working-with-json-schema/images/code-gen.jpg new file mode 100644 index 00000000..25af2bfa Binary files /dev/null and b/content/posts/2022-10-working-with-json-schema/images/code-gen.jpg differ diff --git a/content/posts/2022-10-working-with-json-schema/images/code-gen.png b/content/posts/2022-10-working-with-json-schema/images/code-gen.png deleted file mode 100644 index 9d526315..00000000 Binary files a/content/posts/2022-10-working-with-json-schema/images/code-gen.png and /dev/null differ diff --git a/content/posts/2022-10-working-with-json-schema/images/html-docs.jpg b/content/posts/2022-10-working-with-json-schema/images/html-docs.jpg new file mode 100644 index 00000000..9c6b6162 Binary files /dev/null and b/content/posts/2022-10-working-with-json-schema/images/html-docs.jpg differ diff --git a/content/posts/2022-10-working-with-json-schema/images/html-docs.png b/content/posts/2022-10-working-with-json-schema/images/html-docs.png deleted file mode 100644 index 1087e890..00000000 Binary files a/content/posts/2022-10-working-with-json-schema/images/html-docs.png and /dev/null differ diff --git a/content/posts/2022-10-working-with-json-schema/images/peer-review.jpg b/content/posts/2022-10-working-with-json-schema/images/peer-review.jpg new file mode 100644 index 00000000..5534b19c Binary files /dev/null and b/content/posts/2022-10-working-with-json-schema/images/peer-review.jpg differ diff --git a/content/posts/2022-10-working-with-json-schema/images/peer-review.png b/content/posts/2022-10-working-with-json-schema/images/peer-review.png deleted file mode 100644 index 39e622b4..00000000 Binary files a/content/posts/2022-10-working-with-json-schema/images/peer-review.png and /dev/null differ diff --git a/content/posts/2022-10-working-with-json-schema/images/yml-example.jpg b/content/posts/2022-10-working-with-json-schema/images/yml-example.jpg new file mode 100644 index 00000000..16e024c2 Binary files /dev/null and b/content/posts/2022-10-working-with-json-schema/images/yml-example.jpg differ diff --git a/content/posts/2022-10-working-with-json-schema/images/yml-example.png b/content/posts/2022-10-working-with-json-schema/images/yml-example.png deleted file mode 100644 index 523ea32b..00000000 Binary files a/content/posts/2022-10-working-with-json-schema/images/yml-example.png and /dev/null differ diff --git a/content/posts/2022-10-working-with-json-schema/index.md b/content/posts/2022-10-working-with-json-schema/index.md index ea9f1e4f..fcb01b23 100644 --- a/content/posts/2022-10-working-with-json-schema/index.md +++ b/content/posts/2022-10-working-with-json-schema/index.md @@ -20,7 +20,7 @@ In this post we’ll explore each of these stages in a bit more detail. ## Describing our data formats -{{< figure src="images/yml-example.png" title="A JSON schema document in yml format" >}} +{{
}} We use a shared code repository to describe our data models using JSON schema. These take the form of yml documents, organised around the structure of our product teams, for example Driving Licence, Vehicles or Payments. @@ -36,7 +36,7 @@ We also have a set of test and build scripts that help ensure consistency and qu ## Collaboration and Peer Review -{{< figure src="images/peer-review.png" title="An example of a pull request reviewing a schema" >}} +{{
}} Peer review forms an important part of the process and is an opportunity to review, discuss and improve our data models. This all takes place collaboratively and openly before we write any code, which we have found to really increase efficiency and quality. @@ -49,15 +49,15 @@ This process often generates insightful questions that help develop a clear unde ## Human-friendly documentation -{{< figure src="images/html-docs.png" title="Human friendly documentation published as HTML" >}} +{{
}} -We use the [json-schema-static-docs](https://tomcollins.github.io/json-schema-static-docs/) library to generate markdown versions of our schema definitions. This are published as a static website, using the hugo framework, and made available to developers and other stakeholders. +We use the [json-schema-static-docs](https://tomcollins.github.io/json-schema-static-docs/) library to generate markdown versions of our schema definitions. These are published as a static website, using the hugo framework, and made available to developers and other stakeholders. This has become the source of truth for our conceptual data models and has increased collaboration, consistency and reuse of data models and schema. The open nature of the documentation means that product teams can easily view and understand the data types used by other teams which helps drive out consistency of approach, naming, structure etc. A lot of this can happen before any code or API contracts are created which has increased efficiency. But fear not, this remains an iterative agile process, and is not all done up-front. ## Sharing schema and Open API -As part of our build process we publish an npm package containing yml and json versions our schema to our internal repository. We also generate a de-referenced version of the schema using json-schema-ref-parser as this makes it easier for other tools to consume the schema. +As part of our build process we publish a npm package containing yml and json versions our schema to our internal repository. We also generate a de-referenced version of the schema using json-schema-ref-parser as this makes it easier for other tools to consume the schema. This all allows our teams to use the schema within their processes and applications to do things like: @@ -66,11 +66,11 @@ This all allows our teams to use the schema within their processes and applicati ## Code generation -{{< figure src="images/code-gen.png" title="Typescript interfaces generated fron JSON Schema" >}} +{{
}} Finally, our build process includes code generation steps that creates representations of our data models in Java and TypeScript. This allows teams to import the generated code into their applications to give them strongly typed models, making it easier to work with data and reducing the opportunity for various types of error. -The Java code generation step creates Java classes using the jsonschema2pojo utility which gets published to our internal maven repository. For TypeScript we generate interfaces using json-schema-to-typescript and these get bundled with the schema in an npm package. +The Java code generation step creates Java classes using the jsonschema2pojo utility which gets published to our internal maven repository. For TypeScript, we generate interfaces using json-schema-to-typescript and these get bundled with the schema in a npm package. ## Conclusion diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarm.jpg b/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarm.jpg new file mode 100644 index 00000000..5d7b8917 Binary files /dev/null and b/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarm.jpg differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarm.png b/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarm.png deleted file mode 100644 index be165697..00000000 Binary files a/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarm.png and /dev/null differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarms.jpg b/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarms.jpg new file mode 100644 index 00000000..f45903ff Binary files /dev/null and b/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarms.jpg differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarms.png b/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarms.png deleted file mode 100644 index 6b334a1a..00000000 Binary files a/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible-with-alarms.png and /dev/null differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible.png b/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible.png deleted file mode 100644 index 5908f488..00000000 Binary files a/content/posts/2023-03-improving-our-dead-letter-queues/images/approx-number-of-messages-visible.png and /dev/null differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/aws-sqs-dlq-redrive.jpeg b/content/posts/2023-03-improving-our-dead-letter-queues/images/aws-sqs-dlq-redrive.jpeg deleted file mode 100644 index 3083e01c..00000000 Binary files a/content/posts/2023-03-improving-our-dead-letter-queues/images/aws-sqs-dlq-redrive.jpeg and /dev/null differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/aws-sqs-dlq-redrive.jpg b/content/posts/2023-03-improving-our-dead-letter-queues/images/aws-sqs-dlq-redrive.jpg new file mode 100644 index 00000000..46a0c521 Binary files /dev/null and b/content/posts/2023-03-improving-our-dead-letter-queues/images/aws-sqs-dlq-redrive.jpg differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/cloudwatch-pagerduty-integration.jpg b/content/posts/2023-03-improving-our-dead-letter-queues/images/cloudwatch-pagerduty-integration.jpg new file mode 100644 index 00000000..4372dfaf Binary files /dev/null and b/content/posts/2023-03-improving-our-dead-letter-queues/images/cloudwatch-pagerduty-integration.jpg differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/cloudwatch-pagerduty-integration.png b/content/posts/2023-03-improving-our-dead-letter-queues/images/cloudwatch-pagerduty-integration.png deleted file mode 100644 index f7d4720a..00000000 Binary files a/content/posts/2023-03-improving-our-dead-letter-queues/images/cloudwatch-pagerduty-integration.png and /dev/null differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/queue-with-dlq-and-alarm.jpg b/content/posts/2023-03-improving-our-dead-letter-queues/images/queue-with-dlq-and-alarm.jpg new file mode 100644 index 00000000..6492c330 Binary files /dev/null and b/content/posts/2023-03-improving-our-dead-letter-queues/images/queue-with-dlq-and-alarm.jpg differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/queue-with-dlq-and-alarm.png b/content/posts/2023-03-improving-our-dead-letter-queues/images/queue-with-dlq-and-alarm.png deleted file mode 100644 index 6b0d77ee..00000000 Binary files a/content/posts/2023-03-improving-our-dead-letter-queues/images/queue-with-dlq-and-alarm.png and /dev/null differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-daily-alarm.jpg b/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-daily-alarm.jpg new file mode 100644 index 00000000..a1326c3c Binary files /dev/null and b/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-daily-alarm.jpg differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-daily-alarm.png b/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-daily-alarm.png deleted file mode 100644 index bbcd6005..00000000 Binary files a/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-daily-alarm.png and /dev/null differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-deletes-messages.jpg b/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-deletes-messages.jpg new file mode 100644 index 00000000..dfbd991f Binary files /dev/null and b/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-deletes-messages.jpg differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-deletes-messages.png b/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-deletes-messages.png deleted file mode 100644 index 4ffb6814..00000000 Binary files a/content/posts/2023-03-improving-our-dead-letter-queues/images/sqs-deletes-messages.png and /dev/null differ diff --git a/content/posts/2023-03-improving-our-dead-letter-queues/index.md b/content/posts/2023-03-improving-our-dead-letter-queues/index.md index 23e92631..f1160869 100644 --- a/content/posts/2023-03-improving-our-dead-letter-queues/index.md +++ b/content/posts/2023-03-improving-our-dead-letter-queues/index.md @@ -31,7 +31,7 @@ In a typical setup you may have: 3. A dead-letter queue that can hold messages that can not be processed 4. A means of alerting a human when messages are placed on the DLQ -{{< figure src="images/queue-with-dlq-and-alarm.png" title="DLQ example using AWS" caption="The message handler [2] invokes an external API. If the API is unavailable messages will end up on the DLQ, triggering an alarm." >}} +{{
}} ### You may not be able to process every message @@ -55,7 +55,7 @@ We don't want items appearing on our DLQ as it means something has gone wrong an If your messages were moved to the DLQ because a dependency was temporarily unavailable then you may just need to move the messages back onto the source queue (once the dependency is available) so they can be processed again. -{{< figure src="images/aws-sqs-dlq-redrive.jpeg" title="The DLQ redrive feature as it appears in the AWS console." >}} +{{
}} If you're using AWS, and have the appropriate permissions, source queue re-drive can be achieved [through the AWS console](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue-redrive.html). This capability is not exposed through an API so may be something you need to implement yourself if the console is not an option in a production environment. @@ -73,7 +73,7 @@ Other scenarios and queuing services may require a different approach and you sh ### SQS has a maximum message retention period of 14 days -{{< figure src="images/sqs-deletes-messages.png" >}} +{{
}} We use AWS SQS for both source queues and a dead-letter queues and SQS has a [maximum message retention period](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-basic-architecture.html) of fourteen days. Once a message has sat on a queue for this duration _the message will be deleted_. @@ -128,7 +128,7 @@ Clearly we don't want to lose messages, so we spent some time analysing what wen ### Understanding our DLQ alarm behaviour -{{< figure src="images/approx-number-of-messages-visible-with-alarm.png" caption="Our original alarm threshold was triggered once when messages were initially added to the queue." >}} +{{
}} Messages were added to the DLQ in distinct phases over a period of time but just a single alarm was triggered and a single incident raised. @@ -159,7 +159,7 @@ The alarm threshold was met once at 07:53 and entered an `ALARM` state, but neve We discussed the incident at one of our regular engineering community of practice meetings, both to raise awareness and to try and learn from each other. One of our teams shared their approach: -> if the number of messages added to the DLQ in the last 5 minutes > 0 then raise an alarm +> If the number of messages added to the DLQ in the last 5 minutes > 0 then raise an alarm Which results in the alarm being triggered multiple times: @@ -174,28 +174,28 @@ Which results in the alarm being triggered multiple times: The threshold is met at 07:53 entering an `ALARM` state then returns to `OK` within five minutes. This allows the threshold to crossed again when subsequent messages are added to the queue (08:02, 08:13). Much nicer. -{{< figure src="images/approx-number-of-messages-visible-with-alarms.png" caption="Now the alarm triggers multiple times.">}} +{{
}} If you want to set his up in Cloudwatch it will look something like this: ```yml DLQueueAlarm: - Type: AWS::CloudWatch::Alarm - Properties: - AlarmDescription: 'Message(s) added to the SMS DLQ in the last 5 minutes' - Namespace: 'AWS/SQS' - MetricName: NumberOfMessagesSent - Dimensions: - - Name: QueueName - Value: 'my-dlq' - Statistic: Sum - Period: 60 - EvaluationPeriods: 5 - DatapointsToAlarm: 1 - Threshold: 1 - ComparisonOperator: GreaterThanOrEqualToThreshold - AlarmActions: - - Ref: DLQueueAlarmTopic + Type: AWS::CloudWatch::Alarm + Properties: + AlarmDescription: "Message(s) added to the SMS DLQ in the last 5 minutes" + Namespace: "AWS/SQS" + MetricName: NumberOfMessagesSent + Dimensions: + - Name: QueueName + Value: "my-dlq" + Statistic: Sum + Period: 60 + EvaluationPeriods: 5 + DatapointsToAlarm: 1 + Threshold: 1 + ComparisonOperator: GreaterThanOrEqualToThreshold + AlarmActions: + - Ref: DLQueueAlarmTopic ``` > Update 2022-04-04 - it turns out that this example only works when manually adding messages to your DLQ (which you may do while testing the alarm). You can achieve something equivalent using `RATE(ApproximateNumberOfMessagesVisible)>0` and I'll update this example to reflect this ASAP. @@ -212,7 +212,7 @@ One of our developers came up with a really neat cloudwatch expression that make Which will result in the alarm being triggered at around 7am each day if there are messages on the DLQ, even if no further messages have been added. -{{< figure src="images/sqs-daily-alarm.png" title="Daily DLQ alarm" caption="The expression means that the alarm enters an ALARM state at 7 and returns to OK at 9." >}} +{{
}} _Kudos to [Matthew Lewis](https://www.linkedin.com/in/matthew-lewis-277a7157/) for figuring this out 🙌_ @@ -226,9 +226,9 @@ DLQDailyAlarm: AlarmActions: - arn:aws:sns:eu-west-2:1234567890:alert-topic AlarmDescription: Daily alarm for messages on DLQ - AlarmName: 'dlq-daily-alarm' + AlarmName: "dlq-daily-alarm" Metrics: - - Id: summary + - Id: summary Label: DLD Dead Letter Queues Alarm Expression: IF(HOUR(myDLQ)>7 AND HOUR(myDLQ)<9 AND myDLQ > 0, 1, 0) ReturnData: true @@ -240,21 +240,17 @@ We use PagerDuty to manage incidents and alert engineers. When integrating Cloudwatch alarms with PagerDuty to raise [Alerts](https://support.pagerduty.com/docs/alerts) there are a few nuances to be aware of with this sort of alarm threshold. -**Ok actions and auto-resolving alerts** - -When the number of messages added to the DQL within 5 minutes drops to zero the cloudwatch alarm will return to an `OK` state +**OK actions and auto-resolving alerts:** When the number of messages added to the DQL within 5 minutes drops to zero the cloudwatch alarm will return to an `OK` state. If you integrate a Cloudwatch `OK` action with PagerDuty this will cause the PagerDuty Alert to be auto-resolved and closed before a human has had time to notice and take appropriate action. -**Configure alert correlation** - -To ensure that you raise a new alert each day you will need to configure how PagerDuty performs correlation. If you don't do this you may find that your daily alarms are grouped together under the initial alert. +**Configure alert correlation:** To ensure that you raise a new alert each day you will need to configure how PagerDuty performs correlation. If you don't do this you may find that your daily alarms are grouped together under the initial alert. **Correlate events by:** Make a new incident/alert each time **Derive name from:** Alarm Name -{{< figure src="images/cloudwatch-pagerduty-integration.png" title="Configure your Cloudwatch integration within PagerDuty to ensure a new incident is raised each time your alarm triggers." >}} +{{
}} --- @@ -276,4 +272,4 @@ This could include: Hopefully this article will help you think a little more about how you are using DLQs and the processes you have in place to handle any messages that land on them. -All being well this isn't someything you'll have to do frequently but it may be worth reviewing your approach and what alarms or alerts you have in place. +All being well this isn't something you'll have to do frequently but it may be worth reviewing your approach and what alarms or alerts you have in place. diff --git a/content/posts/2023-05-faker-maker-initialisation/index.md b/content/posts/2023-05-faker-maker-initialisation/index.md index 2ca2459b..41d4ebb0 100644 --- a/content/posts/2023-05-faker-maker-initialisation/index.md +++ b/content/posts/2023-05-faker-maker-initialisation/index.md @@ -12,12 +12,13 @@ TocOpen: true In the world of automation testing, generating realistic-looking test data is a preferred approach to using static fixtures. -[FakerMaker](https://billyruffian.github.io/faker_maker/) is a Ruby gem that allows you to do just that; using factories to create the test data you need. +[FakerMaker](https://billyruffian.github.io/faker_maker/) is a Ruby gem that allows you to do just that; using factories to create the test data you need. It can be used with [Faker](https://github.com/faker-ruby/faker) to dynamically generate test data. ## Initialisation We can use Faker and FakerMaker to create a factory for an API request body like this: + ```ruby require 'faker' require 'faker_maker' @@ -32,12 +33,13 @@ end ``` Which can be called to create a request body like this: + ```ruby body = FM[:example_request].build.as_json -=> {"name"=>"The Hon. Earleen Wunsch", - "buildingNumber"=>"148", - "streetName"=>"Mueller Lodge", - "city"=>"Port Darin", +=> {"name"=>"The Hon. Earleen Wunsch", + "buildingNumber"=>"148", + "streetName"=>"Mueller Lodge", + "city"=>"Port Darin", "country"=>"Canada"} ``` @@ -50,51 +52,61 @@ An example: we need to send a API request with "city" set to "London" and "count This can be done in 4 steps by: 1. Initialising a template request: -```ruby -body = FM[:example_request].build.as_json -=> {"name"=>"Eleni Daniel", - "buildingNumber"=>"50962", - "streetName"=>"Jacqulyn Lights", - "city"=>"New Rubin", - "country"=>"Isle of Man"} -``` + + ```ruby + body = FM[:example_request].build.as_json + => {"name"=>"Eleni Daniel", + "buildingNumber"=>"50962", + "streetName"=>"Jacqulyn Lights", + "city"=>"New Rubin", + "country"=>"Isle of Man"} + ``` + 2. Overwriting the value of "city" in the request: -```ruby -body["city"] = "London" -``` + + ```ruby + body["city"] = "London" + ``` + 3. Overwriting the value of "country" in the request: -```ruby -body["country"] = "United Kingom" -``` + + ```ruby + body["country"] = "United Kingom" + ``` + 4. Sending a request containing the overwritten body: -```ruby -body -=> {"name"=>"Eleni Daniel", - "buildingNumber"=>"50962", - "streetName"=>"Jacqulyn Lights", - "city"=>"London", - "country"=>"United Kingdom"} -``` + + ```ruby + body + => {"name"=>"Eleni Daniel", + "buildingNumber"=>"50962", + "streetName"=>"Jacqulyn Lights", + "city"=>"London", + "country"=>"United Kingdom"} + ``` Though this works, it can be considered "more expensive" than needed, as today I learned from a colleague a simple trick to do this more efficiently in just 2 steps... ## Efficient initialising 1. Initialising the request with the values we need for the relevant parameter: -```ruby -body = - FM[:example_request].build(city: "London", country: "United Kingdom").as_json -``` + + ```ruby + body = + FM[:example_request].build(city: "London", country: "United Kingdom").as_json + ``` + 2. Sending a request containing the efficiently initialised body: -```ruby -body -=> {"name"=>"Msgr. Ernesto Reynolds", -"buildingNumber"=>"942", -"streetName"=>"Goyette Passage", -"city"=>"London", -"country"=>"United Kingdom"} -``` + + ```ruby + body + => {"name"=>"Msgr. Ernesto Reynolds", + "buildingNumber"=>"942", + "streetName"=>"Goyette Passage", + "city"=>"London", + "country"=>"United Kingdom"} + ``` Job done. Initialising our test data was done more efficiently in a single line of code. -In the context of large automation test packs dealing with complex data, this simple trick can be particularly beneficial to improving efficiency. \ No newline at end of file +In the context of large automation test packs dealing with complex data, this simple trick can be particularly beneficial to improving efficiency. diff --git a/content/posts/2023-05-logging-with-herodotus/images/logger_output_in_colour.jpg b/content/posts/2023-05-logging-with-herodotus/images/logger_output_in_colour.jpg new file mode 100644 index 00000000..75580a7f Binary files /dev/null and b/content/posts/2023-05-logging-with-herodotus/images/logger_output_in_colour.jpg differ diff --git a/content/posts/2023-05-logging-with-herodotus/images/logger_output_in_colour.png b/content/posts/2023-05-logging-with-herodotus/images/logger_output_in_colour.png deleted file mode 100644 index bea5ddd1..00000000 Binary files a/content/posts/2023-05-logging-with-herodotus/images/logger_output_in_colour.png and /dev/null differ diff --git a/content/posts/2023-05-logging-with-herodotus/index.md b/content/posts/2023-05-logging-with-herodotus/index.md index 189a6e20..5addf4b4 100644 --- a/content/posts/2023-05-logging-with-herodotus/index.md +++ b/content/posts/2023-05-logging-with-herodotus/index.md @@ -51,7 +51,7 @@ Parallel.map(test_data, in_processes: 2) do |person| end ``` -``` +```shell I, [2023-05-15T13:34:15.480703 #75155] INFO -- : Seeding details about Bob I, [2023-05-15T13:34:15.480834 #75156] INFO -- : Seeding details about Alice I, [2023-05-15T13:34:15.481945 #75156] INFO -- : Insert successful @@ -64,7 +64,7 @@ Now, in this example we could probably have a guess based on the process ids tha ## Corralling our test cases with Correlation Ids -Correlation ids are a common way to keep track of the flow of a given flow through a system, which makes them exactly what we are looking for to solve our current problem. Lets get Herodotus set up in the above example and then we will take a dive into how it works under the bonnet. +Correlation ids are a common way to keep track of the flow of a given flow through a system, which makes them exactly what we are looking for to solve our current problem. Let's get Herodotus set up in the above example, and then we will take a dive into how it works under the bonnet. ```ruby module DatabaseAccess @@ -98,7 +98,7 @@ Parallel.map(test_data, in_processes: 2) do |person| end ``` -``` +```shell [2023-05-15 13:58:43 20e66f0f] INFO -- : Seeding details about Alice [2023-05-15 13:58:43 ab0506e8] INFO -- : Seeding details about Bob [2023-05-15 13:58:43 20e66f0f] INFO -- : Insert successful @@ -168,7 +168,7 @@ end Now, while the above logs are a great starting point, it's also important to allow for a degree of flexibility. As such, Herodotus allows for the following configuration changes to be made: -```ruby +```ruby DVLA::Herodotus.configure do |config| config.system_name = 'person-database-testpack' config.pid = true @@ -180,7 +180,7 @@ We'll have a quick overview of the first two before taking a dive into what is g First up, `system_name` is nice and simple. This allows you to add an overall system name to the logs that are being output. `pid` allows you to mimic the behaviour of the default logger, adding the process id to the output at the start of a log message. If we turn both of these on and re-run our test from before, we get an output that looks like this: -``` +```shell [person-database-testpack 2023-05-15 14:59:27 683dd5bb 94910] INFO -- : Seeding details about Alice [person-database-testpack 2023-05-15 14:59:27 3425d11a 94911] INFO -- : Seeding details about Bob [person-database-testpack 2023-05-15 14:59:27 683dd5bb 94910] INFO -- : Insert successful @@ -193,7 +193,7 @@ First up, `system_name` is nice and simple. This allows you to add an overall sy `merge`, the final thing that can be configured within Herodotus is the most complex. Think back to our original example, but this time imagine that rather than the database being configured in a different module, it is something we are pulling in from an external package that also implements Herodotus. Let's have a look at what that would output: -``` +```shell [person-database-testpack 2023-05-15 15:13:01 9e5ca313 98217] INFO -- : Seeding details about Alice [person-database-testpack 2023-05-15 15:13:01 d6812ffc 98218] INFO -- : Seeding details about Bob [database-gem 2023-05-15 15:13:01 fefcf70f 98217] INFO -- : Insert successful @@ -216,11 +216,11 @@ def merge_correlation_ids(new_scenario: nil) end ``` -Now, the above looks complex but if we break it down it's actually quite simple. First up, we are grabbing every instance of `HerodotusLogger` that currently exists by using the [ObjectSpace.each_object](https://ruby-doc.org/core-2.6.1/ObjectSpace.html#method-c-each_object) method, which will return an enumerator that we can use to iterate through all of them. While we are iterating through them, we make sure we don't try and merge this logger with itself, as that can lead to some nasty unexpected behaviour. Once the code is happy it has hold of a different logger, it first stops that logger from trying to merge with the others if has is set up to do so. Again, this is for safety as if that wasn't toggled off we'd end up in an endless loop. What we actually want is a single source of truth, which will be the logger highest up the stack, in our case the one that is created in the tests. Finally, we override that loggers collection of correlation ids and then get it to set itself to the current scenario, safe in the knowledge it will have a correlation id for this scenario as we've just given it that. +Now, the above looks complex but if we break it down it's actually quite simple. First up, we are grabbing every instance of `HerodotusLogger` that currently exists by using the [ObjectSpace.each_object](https://ruby-doc.org/core-2.6.1/ObjectSpace.html#method-c-each_object) method, which will return an enumerator that we can use to iterate through all of them. While we are iterating through them, we make sure we don't try and merge this logger with itself, as that can lead to some nasty unexpected behaviour. Once the code is happy it has hold of a different logger, it first stops that logger from trying to merge with the others if has is set up to do so. Again, this is for safety as if that wasn't toggled off we'd end up in an endless loop. What we actually want is a single source of truth, which will be the logger highest up the stack, in our case the one that is created in the tests. Finally, we override that loggers collection of correlation ids and then get it to set itself to the current scenario, safe in the knowledge it will have a correlation id for this scenario as we've just given it that. Taking that into account and enabling merge on our logger, we get the following output: -``` +```shell [person-database-testpack 2023-05-15 15:29:47 594053ed 4415] INFO -- : Seeding details about Bob [person-database-testpack 2023-05-15 15:29:47 2472e136 4414] INFO -- : Seeding details about Alice [database-gem 2023-05-15 15:29:47 594053ed 4415] ERROR -- : Database rejected insertion, constraint violation @@ -235,8 +235,8 @@ And there we have it, we are merging our ids across different instances of Herod Right back at the start, I said that one of the most important things about these sorts of logs is that they should be easy for a human to read. As such, there is one last thing included in Herodotus that aids in that goal. There are [a collection of extensions to the default String class](https://github.com/dvla/herodotus/blob/main/lib/dvla/herodotus/string.rb) that allow you to simple apply colour to string, helping you easily highlight specific points of interest in your log messages. With a quick pass of colour to our loggers, we can quite quickly get something that looks like this: -{{< figure src="images/logger_output_in_colour.png" title="Logger output, now in colour" >}} +{{
}} ## Conclusion -So. that's the story of how we came up with Herodotus. It's by no means the most complex gem out there, but that's no bad thing. We were looking for something pretty lightweight to extend some already quite powerful functionality within the default logger, just in a way that makes it easier for us to follow the thread through when we are perusing our logs. We've certainly hit that goal and have even come up with some extra functionality along the way to help with running multiple instances of Herodotus in a way that abstracts that complexity away from the consumer. \ No newline at end of file +So. that's the story of how we came up with Herodotus. It's by no means the most complex gem out there, but that's no bad thing. We were looking for something pretty lightweight to extend some already quite powerful functionality within the default logger, just in a way that makes it easier for us to follow the thread through when we are perusing our logs. We've certainly hit that goal and have even come up with some extra functionality along the way to help with running multiple instances of Herodotus in a way that abstracts that complexity away from the consumer. diff --git a/content/posts/2023-05-testing-standard-out-in-ruby/index.md b/content/posts/2023-05-testing-standard-out-in-ruby/index.md index d4c3e089..1e5a5eb3 100644 --- a/content/posts/2023-05-testing-standard-out-in-ruby/index.md +++ b/content/posts/2023-05-testing-standard-out-in-ruby/index.md @@ -17,7 +17,7 @@ Testing what a Ruby process writes to STDOUT. Ruby has two built-in values that represent the system's standard output stream: 1. The constant `STDOUT` -1. The global variable `$stdout` +2. The global variable `$stdout` The Ruby [documentation](https://docs.ruby-lang.org/en/master/globals_rdoc.html), describes `STDOUT` as _the_ standard output, and `$stdout` and the _current_ standard output. If you want to change the stream that this Ruby process sends its output to, you can change the value of `$stdout` @@ -36,4 +36,4 @@ RSpec.describe DVLA::Engineering do puts 'Hello from the DVLA' expect(output.string).to match(/DVLA/) end -``` \ No newline at end of file +``` diff --git a/content/posts/2023-06-efficient-cucumber-step-regex-matching/index.md b/content/posts/2023-06-efficient-cucumber-step-regex-matching/index.md index cc1d7c9a..bb66d4c4 100644 --- a/content/posts/2023-06-efficient-cucumber-step-regex-matching/index.md +++ b/content/posts/2023-06-efficient-cucumber-step-regex-matching/index.md @@ -27,9 +27,10 @@ Given a message of 'I like Cucumber' was added to the queue ``` The step can be captured using the following Cucumber Expression in the step definition block: + ```ruby Given('a message of {string} was added to the queue') do |message_text| - ... + # ... end ``` @@ -41,7 +42,7 @@ If we want to reuse the step definition to perform the same action of adding ano ```ruby Given('a/another message of {string} was added to the queue') do |message_text| - ... + # ... end ``` @@ -64,7 +65,7 @@ Let's attempt to capture the above examples using the following regex in the ste ```ruby Given(/^(a|another) message of '(.*)' was added to the queue$/) do |_unused, message_text| - ... + # ... end ``` @@ -74,21 +75,21 @@ Note that we need the single quotes around the `(.*)` to capture only the string Here's a quick summary of some common regex patterns: -| Patterns | Definition | -|:-----------:|:-------------------------------------------------------------------:| -| `.*` | matches any character (except for newline) between 0 or more times” | -| `.+` | matches at least one of any character (except for line terminators) | -| `.?` | matches one or zero of any character (except for line terminators) | -| `^` | asserts position at start of a line | -| `$` | asserts position at end of a line | -| `\d` | matches a digit (equivalent to [0-9]) | -| `[A-Za-z]*` | matches letter 'A' to 'Z' or 'a' to 'z' between 0 or more times | +| Patterns | Definition | +| ----------- | ------------------------------------------------------------------- | +| `.*` | matches any character (except for newline) between 0 or more times” | +| `.+` | matches at least one of any character (except for line terminators) | +| `.?` | matches one or zero of any character (except for line terminators) | +| `^` | asserts position at start of a line | +| `$` | asserts position at end of a line | +| `\d` | matches a digit (equivalent to [0-9]) | +| `[A-Za-z]*` | matches letter 'A' to 'Z' or 'a' to 'z' between 0 or more times | To simplify this regex better, we can update it to the following: ```ruby Given(/message of '(.*)' was added to the queue$/) do |message_text| - ... + # ... end ``` @@ -116,7 +117,7 @@ We can be more specific about the format of the date by using the following rege ```ruby Given(/date of '([\d]{4}-{\d}{2}-{\d}{2})' was added to the schedule$/) do |date| - ... + # ... end ``` @@ -182,7 +183,7 @@ For example, let's consider the following steps: Scenario: Checkout apples in basket Given there are 5 discounted apples in the basket ... - + Scenario: Checkout medicine in basket Given there are 2 cough medicines in the basket ... diff --git a/content/posts/2023-06-faker-maker-chaos-update/index.md b/content/posts/2023-06-faker-maker-chaos-update/index.md index bce2f633..3c04485d 100644 --- a/content/posts/2023-06-faker-maker-chaos-update/index.md +++ b/content/posts/2023-06-faker-maker-chaos-update/index.md @@ -10,9 +10,9 @@ ShowToc: true TocOpen: true --- -**TL:DR Level up your testing by introducing some chaos to your test data.** +**TL;DR Level up your testing by introducing some chaos to your test data.** -FakerMaker is an incredible factory builder used to generate test data. +FakerMaker is an incredible factory builder used to generate test data. When introduced to a test-pack (alongside Faker), it elevates tests by introducing dynamic test data to ensure we're not constantly testing with the same old boring static data. @@ -26,27 +26,27 @@ FakerMaker.factory :some_fancy_object, naming: :json do some_optional_attribute_4 { Faker::Lorem.word } end -FakerMaker[:some_fancy_object].build.as_json +FakerMaker[:some_fancy_object].build.as_json # => {"someRequiredAttribute"=>"Nulla omnis dolore enim.", "someOptionalAttribute1"=>"qui", "someOptionalAttribute2"=>"deserunt", "someOptionalAttribute3"=>"rerum", "someOptionalAttribute4"=>"facilis"} -FakerMaker[:some_fancy_object].build.as_json +FakerMaker[:some_fancy_object].build.as_json # => {"someRequiredAttribute"=>"Aut repellendus ut quod.", "someOptionalAttribute1"=>"consequatur", "someOptionalAttribute2"=>"quod", "someOptionalAttribute3"=>"pariatur", "someOptionalAttribute4"=>"rerum"} ``` -This is a super effective way of ensuring data within an attribute is dynamic, but can we take this further? +This is a super effective way of ensuring data within an attribute is dynamic, but can we take this further? ## Chaos Engineering -Chaos engineering is a concept that has been around since the early 2000s. The idea is simple - can we build resilient, -fault-tolerant services and how do we prove this? +Chaos engineering is a concept that has been around since the early 2000s. The idea is simple - can we build resilient, +fault-tolerant services and how do we prove this? How does a service cope when it loses connection to a database? -What about a service that has gone down that our service depends on? +What about a service that has gone down that our service depends on? To prove a system is resilient against these types of issues we might experiment and replicate these issues to monitor -the impact. Remove the database connection, terminate a dependent service etc. +the impact. Remove the database connection, terminate a dependent service etc. + +In general, cause chaos. So let's take that concept, and apply it to something like test data! -In general, cause chaos. So let's take that concept, and apply it to something like test data! - ## Introducing Chaos for FakerMaker FakerMaker v2 introduces chaos mode. @@ -62,19 +62,19 @@ FakerMaker.factory :some_fancy_object, naming: :json do some_optional_attribute_3(optional: 10) { Faker::Lorem.word } end -FakerMaker[:some_fancy_object].build(chaos: true).as_json +FakerMaker[:some_fancy_object].build(chaos: true).as_json # => {"someRequiredAttribute1"=>"Sed rem magnam placeat.", "someRequiredAttribute2"=>"nemo", "someOptionalAttribute2"=>"dolores"} -FakerMaker[:some_fancy_object].build(chaos: true).as_json +FakerMaker[:some_fancy_object].build(chaos: true).as_json # => {"someRequiredAttribute1"=>"Hic neque aut sapiente.", "someRequiredAttribute2"=>"saepe", "someOptionalAttribute1"=>"id"} -FakerMaker[:some_fancy_object].build(chaos: true).as_json +FakerMaker[:some_fancy_object].build(chaos: true).as_json # => {"someRequiredAttribute1"=>"Fuga est rerum quia.", "someRequiredAttribute2"=>"vero", "someOptionalAttribute2"=>"nulla", "someOptionalAttribute1"=>"dolorem"} ``` -When a factory is built with chaos enabled, optional fields become truly optional and may not be present in the built factory. +When a factory is built with chaos enabled, optional fields become truly optional and may not be present in the built factory. Not only is the data dynamic per attribute, but the attributes themselves are now dynamic ... chaos! -Lets dig in a bit ... +Let's dig in a bit ... ### Attribute metadata @@ -84,6 +84,7 @@ Attributes can now be tagged as `required` or `optional`. some_required_attribute_1(required: true) { Faker::Lorem.sentence } some_optional_attribute_1(optional: true) { Faker::Lorem.sentence } ``` + > By default, all attributes are `optional`. You can also pass through an optional weighting. This determines the likelihood of that attribute appearing in your built factory @@ -92,13 +93,14 @@ You can also pass through an optional weighting. This determines the likelihood some_optional_attribute_1(optional: 90) { Faker::Lorem.sentence } # 90% chance this attribute will be present some_optional_attribute_2(optional: 10) { Faker::Lorem.sentence } # 10% chance this attribute will be present ``` + > By default, all `optional` attributes are weighted at `50%`. -Metadata on attributes provides some benefits without chaos enabled. +Metadata on attributes provides some benefits without chaos enabled. -Factories are often modelling schemas listed in API contracts. These contracts state which fields are required. +Factories are often modelling schemas listed in API contracts. These contracts state which fields are required. -By having this data available on the factory it gives us at a glance information at the data we are modeling without +By having this data available on the factory it gives us at a glance information at the data we are modeling without the hassle of checking API contracts. ### Enabling Chaos @@ -114,20 +116,21 @@ You can also specify which fields to run chaos over by passing an `Array`. ```ruby FakerMaker[:some_fancy_object].build(chaos: %i[some_optional_attribute_1]) ``` -> Chaos will only run on `some_optional_attribute_1` and not against the other optional attributes. +> Chaos will only run on `some_optional_attribute_1` and not against the other optional attributes. ### Why would I want this? -By using this approach to testing, we are more closely mimicking real life scenarios. +By using this approach to testing, we are more closely mimicking real life scenarios. -Data comes in all shapes and sizes, especially when optional attributes come into play. +Data comes in all shapes and sizes, especially when optional attributes come into play. How do you know an APIs optional field is truly optional? How do you know your beautiful webpage that displays data with optional fields will actually work? Chaos introduces new combinations of tests without the hassle or mess of doing it yourself. ### TL:DR + - Every attribute is `optional` by default - Optional attributes maybe removed when your factory is built and chaos is enabled - Attributes can be marked as `required` - required attributes will always be present diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-1.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-1.jpg new file mode 100644 index 00000000..b2751464 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-1.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-1.png b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-1.png deleted file mode 100644 index 6d3f0191..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-1.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-2.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-2.jpg new file mode 100644 index 00000000..859a2799 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-2.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-2.png b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-2.png deleted file mode 100644 index 2694f65e..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-2.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-3.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-3.jpg new file mode 100644 index 00000000..135c57d2 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-3.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-3.png b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-3.png deleted file mode 100644 index cd495e6e..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-3.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-4.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-4.jpg new file mode 100644 index 00000000..aae39873 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-4.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-4.png b/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-4.png deleted file mode 100644 index 2e269022..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Chrome-Step-4.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-1.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-1.jpg new file mode 100644 index 00000000..71ac8b2e Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-1.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-1.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-1.png deleted file mode 100644 index 1fe6c7b1..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-1.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-2.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-2.jpg new file mode 100644 index 00000000..63573992 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-2.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-2.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-2.png deleted file mode 100644 index c8ece701..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-2.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-3.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-3.jpg new file mode 100644 index 00000000..f8aa4a42 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-3.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-3.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-3.png deleted file mode 100644 index ba0feb5c..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-3.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-4.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-4.jpg new file mode 100644 index 00000000..12715736 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-4.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-4.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-4.png deleted file mode 100644 index ca6b2911..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-2-Step-4.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-1.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-1.jpg new file mode 100644 index 00000000..6d76f527 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-1.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-1.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-1.png deleted file mode 100644 index 453f37cc..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-1.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-2.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-2.jpg new file mode 100644 index 00000000..f61738af Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-2.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-2.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-2.png deleted file mode 100644 index abe32fdb..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-2.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.1.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.1.jpg new file mode 100644 index 00000000..384cc79b Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.1.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.1.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.1.png deleted file mode 100644 index 93c878ec..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.1.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.jpg new file mode 100644 index 00000000..311efbde Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.png deleted file mode 100644 index 758d6ed6..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-3.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-4.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-4.jpg new file mode 100644 index 00000000..6f752ed8 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-4.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-4.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-4.png deleted file mode 100644 index 87fd1811..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-4.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-5.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-5.jpg new file mode 100644 index 00000000..f20e6cdd Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-5.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-5.png b/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-5.png deleted file mode 100644 index 9aa75fd6..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Firefox-Step-5.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-1.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-1.jpg new file mode 100644 index 00000000..f996cf92 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-1.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-1.png b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-1.png deleted file mode 100644 index a981febe..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-1.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-2.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-2.jpg new file mode 100644 index 00000000..fc6551ed Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-2.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-2.png b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-2.png deleted file mode 100644 index 7c4c92fa..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-2.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-3.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-3.jpg new file mode 100644 index 00000000..da7d09a4 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-3.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-3.png b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-3.png deleted file mode 100644 index 86f9d530..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-3.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-4.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-4.jpg new file mode 100644 index 00000000..e4db5753 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-4.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-4.png b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-4.png deleted file mode 100644 index 9694f905..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-4.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-5.jpg b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-5.jpg new file mode 100644 index 00000000..445e6567 Binary files /dev/null and b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-5.jpg differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-5.png b/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-5.png deleted file mode 100644 index 0b1b48b6..00000000 Binary files a/content/posts/2023-07-taking-full-length-screenshots/images/Safari-Step-5.png and /dev/null differ diff --git a/content/posts/2023-07-taking-full-length-screenshots/index.md b/content/posts/2023-07-taking-full-length-screenshots/index.md index 517657e0..f3ddd4ea 100644 --- a/content/posts/2023-07-taking-full-length-screenshots/index.md +++ b/content/posts/2023-07-taking-full-length-screenshots/index.md @@ -10,7 +10,7 @@ ShowToc: true TocOpen: true --- -**TL:DR Capture full-length webpage screenshots using built in Developer Tools.** +**TL;DR Capture full-length webpage screenshots using built in Developer Tools.** Sometimes when developing a webpage you may need to send screenshots back and forth between members of your squad or business. @@ -22,85 +22,107 @@ In order to do this you will need to firstly open your browser of choice and nav Both Chrome and Edge use the same shortcuts and developer tools. So follow the steps below if you are a Chrome or Edge user. -1. Once on the page you would like to capture you should firstly access Developer Tools by clicking `Cmd` + `Opt` + `I` *(on Mac)* or `Ctrl` + `Shift` + `I` *(on Windows)* +1. Once on the page you would like to capture you should firstly access Developer Tools by clicking `Cmd` + `Opt` + `I` _(on Mac)_ or `Ctrl` + `Shift` + `I` _(on Windows)_ -![Chrome-Step-1.png](images%2FChrome-Step-1.png) + {{
}} -2. Then, press `Cmd` + `shift` + `P` *(on Mac)* or `Ctrl` + `Shift` + `P` *(on Windows)*. +2. Then, press `Cmd` + `shift` + `P` _(on Mac)_ or `Ctrl` + `Shift` + `P` _(on Windows)_. -![Chrome-Step-2.png](images%2FChrome-Step-2.png) + {{
}} 3. In the search bar, immediately after the word Run >, type "screenshot". -![Chrome-Step-3.png](images%2FChrome-Step-3.png) + {{
}} -4. Select 'Capture full size screenshot', and Chrome/Edge will automatically save a full-page screenshot to your Downloads folder as a png and it will generate the name based off the webpage's URL. +4. Select 'Capture full size screenshot', and Chrome/Edge will automatically save a full-page screenshot to your Downloads folder as a png, and it will generate the name based off the webpage's URL. -![Chrome-Step-4.png](images%2FChrome-Step-4.png) + {{
}} ## Firefox -With Firefox there are two methods to allow you take to take full size screenshots, one is with Developer Tools and the other is with a simple right click. We will go through both methods below and you can decide which you prefer. -1. Firstly navigate to the page you would like to capture, and then access Developer Tools by clicking `Cmd` + `Opt` + `I` *(on Mac)* or `Ctrl` + `Shift` + `I` *(on Windows)* -![Firefox-Step-1.png](images%2FFirefox-Step-1.png) +With Firefox there are two methods to allow you take to take full size screenshots, one is with Developer Tools and the other is with a simple right click. We will go through both methods below, and you can decide which you prefer. + +1. Firstly navigate to the page you would like to capture, and then access Developer Tools by clicking `Cmd` + `Opt` + `I` _(on Mac)_ or `Ctrl` + `Shift` + `I` _(on Windows)_ + + {{
}} 2. Next you will want to open the developer tools setting menu by clicking on the three dots on the top right of the dev tools menu -![Firefox-Step-2.png](images%2FFirefox-Step-2.png) + + {{
}} 3. Once this is opened you will want to enable the "Take a screenshot of the entire page" option by clicking the tick box in "Available Toolbox Buttons" -![Firefox-Step-3.png](images%2FFirefox-Step-3.png) -![Firefox-Step-3.1.png](images%2FFirefox-Step-3.1.png) -4. Once that is enabled you will now have access to the *"Take a screenshot of the entire page"* (the small camera) button. - ![Firefox-Step-4.png](images%2FFirefox-Step-4.png) + {{
}} + + {{
}} + +4. Once that is enabled you will now have access to the _"Take a screenshot of the entire page"_ (the small camera) button. + + {{
}} 5. When the button is clicked it will capture a screenshot of the entire page and will save it as a png to your downloads folder using the date and time the screenshot was taken followed by '-fullpage' as the name. - ![Firefox-Step-4.1.png](images%2FFirefox-Step-5.png) - -### Second method + + {{
}} + +### Second method The second method is a lot simpler and doesn't require you to access the developer tools. **It also allows you to just copy the full page screenshot if you don't want to download it to your device.** -1. Once on the page you would like to capture, you can right click and select 'Take Screenshot' -![Firefox-2-Step-1.png](images%2FFirefox-2-Step-1.png) -2. Next the screen will go dark and you will be given the option to drag and select the region you would like to capture. But to capture the full screen you must click on the 'Save full page' button -![Firefox-2-Step-2.png](images%2FFirefox-2-Step-2.png) +1. Once on the page you would like to capture, you can right-click and select 'Take Screenshot' + + {{
}} + +2. Next the screen will go dark, and you will be given the option to drag and select the region you would like to capture. But to capture the full screen you must click on the 'Save full page' button + + {{
}} + 3. After the button is clicked it will display the screenshot and give you the option to either download or just copy the screenshot -![Firefox-2-Step-3.png](images%2FFirefox-2-Step-3.png) + + {{
}} + 4. If you choose to download it then it will get saved to your downloads folder as a png using the current date and time followed by the page's title as the screenshots name -![Firefox-2-Step-4.png](images%2FFirefox-2-Step-4.png) + + {{
}} + ## Safari -For Safari it requires a few extra steps, such as enabling the developer console. + +For Safari, it requires a few extra steps, such as enabling the developer console. ### How to enable Developer menu in safari + 1. Firstly you will need to navigate to safari's settings. This can be located by clicking the safari tab and selecting settings. Or by simply clicking `Cmd` + `,` -![Safari-Step-1.png](images%2FSafari-Step-1.png) - + + {{
}} + 2. Next you will need to navigate to the "Advanced" tab in safari's settings and then enable "Show Develop menu in menu bar". Once this is ticked you will now have access to the Developer menu in safari. -![Safari-Step-2.png](images%2FSafari-Step-2.png) + + {{
}} ### How to capture a screenshot in safari + Once you have got the developer console set up then you should be ready to capture a full page screenshot. 1. Once on the page you would like to capture you should firstly access Developer Tools by clicking `Cmd` + `Opt` + `I` -![Safari-Step-3.png](images%2FSafari-Step-3.png) + + {{
}} 2. `Control-click` or `right-click` while hovering over the **< html >** tag. You’ll get a flyout menu, within which you can select **Capture Screenshot**. -![Safari_Step-4.png](images%2FSafari-Step-4.png) + + {{
}} 3. Once clicked you will then be offered the choice of where you would like to save the screenshot and what you would like to name it. -![Safari-Step-5.png](images%2FSafari-Step-5.png) + {{
}} ## Why would I want this? -It is important to communicate updates that you are producing to your webpages. Often when you go to take a screenshot using your device you can only capture what is available on your screen at the time which, *unless you have a very large monitor*, may not capture all the information on the page. + +It is important to communicate updates that you are producing to your webpages. Often when you go to take a screenshot using your device you can only capture what is available on your screen at the time which, _unless you have a very large monitor_, may not capture all the information on the page. Also when creating pull requests it is nice to include a screenshot of the changes you have implemented so that the approvers can view them without needing to spin up the changes locally. ### TL:DR - You can capture full page screenshots in **Chrome, Edge, Firefox** and **Safari** using dev tools. -- To access Developer Tools *(if enabled)* you can use the shortcut `Cmd` + `Opt` + `I` *(on Mac)* or `Ctrl` + `Shift` + `I` *(on Windows)* +- To access Developer Tools _(if enabled)_ you can use the shortcut `Cmd` + `Opt` + `I` _(on Mac)_ or `Ctrl` + `Shift` + `I` _(on Windows)_ - Chrome and Edge use the same screenshotting method, Firefox allows you to copy rather than download the screenshot if you choose and Safari allows you to name and select where the screenshot gets saved. ---- diff --git a/content/posts/2023-08-at-exit-hook/index.md b/content/posts/2023-08-at-exit-hook/index.md index a06b7e31..5e7802d5 100644 --- a/content/posts/2023-08-at-exit-hook/index.md +++ b/content/posts/2023-08-at-exit-hook/index.md @@ -10,9 +10,10 @@ ShowToc: true TocOpen: true --- -Traditionally, in functional testing, a clean up/tear down script would be run in something called an `After` hook. These hooks are part of the Cucumber DSL and are designed to execute when a scenario has finished. +Traditionally, in functional testing, a clean-up/tear down script would be run in something called an `After` hook. These hooks are part of the Cucumber DSL and are designed to execute when a scenario has finished. Cucumber example: + ```ruby After do |scenario| if scenario.failed? @@ -21,9 +22,9 @@ After do |scenario| end ``` -There is a drawback to these hooks where they will fail to run when certain exit codes are returned from the scenario or the program is interrupted. This isn't great when you need these scripts to execute on every run, regardless of the exit reason. +There is a drawback to these hooks where they will fail to run when certain exit codes are returned from the scenario or the program is interrupted. This isn't great when you need these scripts to execute on every run, regardless of the exit reason. -This is where `at_exit` comes in. +This is where `at_exit` comes in. ## The at_exit function @@ -34,7 +35,7 @@ to consider when using it. It works by converting a given block to a [Proc](https://ruby-doc.org/core-3.0.0/Proc.html) and registers it for execution when the program exits. -## Example: +## Example Here's how this function is used in the TYR project to clear our Mailsac account of all temporary inboxes created within the tests. diff --git a/content/posts/2023-08-calling-fakermaker-to-generate-numerous-objects/index.md b/content/posts/2023-08-calling-fakermaker-to-generate-numerous-objects/index.md index 3945247c..40166482 100644 --- a/content/posts/2023-08-calling-fakermaker-to-generate-numerous-objects/index.md +++ b/content/posts/2023-08-calling-fakermaker-to-generate-numerous-objects/index.md @@ -12,13 +12,14 @@ TocOpen: true Trying to generate numerous, realistic-looking objects for automation testing can be difficult. -Using [FakerMaker](https://billyruffian.github.io/faker_maker/) to do just that is preferable; using factories with [Faker](https://github.com/faker-ruby/faker) to dynamically create the test data you need in whatever format you need, is far better than using fixtures. +Using [FakerMaker](https://billyruffian.github.io/faker_maker/) to do just that is preferable; using factories with [Faker](https://github.com/faker-ruby/faker) to dynamically create the test data you need in whatever format you need, is far better than using fixtures. But what if you needed more than one object of the particular factory that you have created? ## Creating a factory We can use Faker and FakerMaker to create an individual factory for an API request body like this: + ```ruby require 'faker' require 'faker_maker' @@ -35,27 +36,30 @@ FakerMaker.factory :grocery_order do grocery_total(json: 'groceryTotal', omit: :nil) {Faker::Number.number(digits: 6).to_s } end ``` -But what if the test requires MORE THAN ONE instance of this fairly complex object be generated? +But what if the test requires MORE THAN ONE instance of this fairly complex object be generated? ## Creating multiple instances of the FM factory object This can be done in 2 ways: -1. Using Array.new: -```ruby -multiple_orders = Array.new(3) { FM[:grocery_order].build.as_json } -``` -This will create a new array of length 3 AND build 3 instances of the grocery_order object. The multiple_orders array can then be returned for your tests. +1. Using `Array.new`: -2. Using .map: -```ruby -multiple_orders = 3.times.map { FM[:grocery_order].build.as_json } -``` -This will do the exact same thing as the above. This is a more elegant way of writing the above code, and is more readable, in my opinion. + ```ruby + multiple_orders = Array.new(3) { FM[:grocery_order].build.as_json } + ``` + + This will create a new array of length 3 AND build 3 instances of the grocery_order object. The multiple_orders array can then be returned for your tests. +2. Using `.map`: + + ```ruby + multiple_orders = 3.times.map { FM[:grocery_order].build.as_json } + ``` + +This will do the exact same thing as the above. This is a more elegant way of writing the above code, and is more readable, in my opinion. -And that is it! +And that is it! ## Conclusion diff --git a/content/posts/2023-11-artefacts-and-atlas/index.md b/content/posts/2023-11-artefacts-and-atlas/index.md index 90df8542..b5ac2226 100644 --- a/content/posts/2023-11-artefacts-and-atlas/index.md +++ b/content/posts/2023-11-artefacts-and-atlas/index.md @@ -10,7 +10,7 @@ ShowToc: true TocOpen: true --- -We've recently publicly released a Gem called [dvla-atlas](https://github.com/dvla/atlas)[^1] and today we are going to take you through a bit of history surrounding testing at the DVLA that led to use developing Atlas, along with a dive into some of the code that makes it tick. Atlas is designed to make the managing of properties in functional tests easier while also ensuring that each test is run in isolation and without any cross-pollination of test data. But first, lets take a look at what we mean by artefacts and we used them in tests we'd written in [Cucumber](https://cucumber.io/). +We've recently publicly released a Gem called [dvla-atlas](https://github.com/dvla/atlas)[^1], and today we are going to take you through a bit of history surrounding testing at the DVLA that led to use developing Atlas, along with a dive into some of the code that makes it tick. Atlas is designed to make the managing of properties in functional tests easier while also ensuring that each test is run in isolation and without any cross-pollination of test data. But first, lets take a look at what we mean by artefacts, and we used them in tests we'd written in [Cucumber](https://cucumber.io/). [^1]: As you'll see, we've named it that because it supports the `World` @@ -18,7 +18,7 @@ We've recently publicly released a Gem called [dvla-atlas](https://github.com/dv We follow Acceptance Test Driven Design (ATDD) wherever we can and use Cucumber and Gherkin as the backbone for expressing customer wants in a specification we can use to see if we've actually met that want. Oh, and we love ruby. -``` +```gherkin Given I am using Cucumber When I write a specification Then I will have a series of steps @@ -26,7 +26,7 @@ Then I will have a series of steps With the specification above, we have three steps with a clear arrange-act-assert order which collectively forms one scenario -- one test. On the programmatic side, these are modelled as three separate methods and to pass information from one to the next we use instance variables. -Since we're using a dynamic language we can pop these into existance at will. This is fine but has two problems: +Since we're using a dynamic language we can pop these into existence at will. This is fine but has two problems: 1. Using very generic variable names 2. Typos @@ -41,9 +41,9 @@ expect(@badly_spelled_instance_variable).to be_nil ...will always succeed if we've misspelled our instance variable's name. -To mitigate this problem, we've adopted a stategy of declaring upfront those bits of state we want to keep a hold of and pass from one step to another. We built a `Struct` in which we could declare the fields we wanted and a helper method we called `artefacts`. Declaring this upgront meant we thought quite hard about what to name these things and more meaning emerged. The `Struct` helped us catch misspelt fields by throwing exceptions at runtime if we used a non-existent field. +To mitigate this problem, we've adopted a strategy of declaring upfront those bits of state we want to keep a hold of and pass from one step to another. We built a `Struct` in which we could declare the fields we wanted and a helper method we called `artefacts`. Declaring this upfront meant we thought quite hard about what to name these things and more meaning emerged. The `Struct` helped us catch misspelt fields by throwing exceptions at runtime if we used a non-existent field. -Our needs grew though. We'd bolt [FakerMaker](https://billyruffian.github.io/faker_maker/) factory built objects in here with default values. `Struct` wasn't cutting it for us so we switched to full fat classes with neat control over object construction and defaults. This was transparent to us because we always used our lovely convenice method... +Our needs grew though. We'd bolt [FakerMaker](https://billyruffian.github.io/faker_maker/) factory built objects in here with default values. `Struct` wasn't cutting it for us so we switched to full fat classes with neat control over object construction and defaults. This was transparent to us because we always used our lovely convenience method... ```ruby def artefacts @@ -55,9 +55,9 @@ end artefacts.my_field ``` -Which breaks down as soon as you start grouping functionality into modules or classes because, if you're not very careful, you can end up with two (or more) distinct instances of `@artefacts` with different scopes. Singleton-patterm instances also don't fit our needs because we need to throw away all of the artefacts after each _scenario_ and can't have them leaking state between each other. +Which breaks down as soon as you start grouping functionality into modules or classes because, if you're not very careful, you can end up with two (or more) distinct instances of `@artefacts` with different scopes. Singleton-pattern instances also don't fit our needs because we need to throw away all of the artefacts after each _scenario_ and can't have them leaking state between each other. -So this is problem we want to solve: +So this is problem we want to solve: 1. a place to define the state we want to share up front 2. to have the runtime throw an error if some undefined state is referenced @@ -65,11 +65,11 @@ So this is problem we want to solve: 4. we want it to be super easy to use 5. work transparently with bare methods and those in modules and classes -So we wrote Altas. It's a small library, but a great deal of thought has gone into it and make it hit each of those needs. +So we wrote Atlas. It's a small library, but a great deal of thought has gone into it and make it hit each of those needs. ## Atlas -With Atlas, we hope to formalise what we were doing with artefacts without distrupting the way we like to write tests. Our goal was to leverage some functionality built into Cucumber called `World`[^2] while also having something nice and easy to pick up and use. +With Atlas, we hope to formalise what we were doing with artefacts without disrupting the way we like to write tests. Our goal was to leverage some functionality built into Cucumber called `World`[^2] while also having something nice and easy to pick up and use. [^2]: If you haven't come across `World` in Cucumber before, it's a way of influencing the context within with a test scenario is run. You don't need to worry about the specifics for this post, as we'll explain them as they become relevant, but if you are interested in the inner workings you can find some tests that document the common uses for it in [the Ruby implementation of Cucumber](https://github.com/cucumber/cucumber-ruby/blob/main/features/docs/writing_support_code/world.feature) @@ -97,7 +97,7 @@ World do end ``` -With the above set up, you'd be able to access `value` and call `add_to_value` in any test step, with that block being called again at the beginning of each new scenario to generate the context that it will be run in. Therefore, each test will start with a `value` of 2. It's on top of this Cucumber functionality that we built Atlas: +With the above set-up, you'd be able to access `value` and call `add_to_value` in any test step, with that block being called again at the beginning of each new scenario to generate the context that it will be run in. Therefore, each test will start with a `value` of 2. It's on top of this Cucumber functionality that we built Atlas: ```ruby World do @@ -121,7 +121,7 @@ World do end ``` -That gives all test steps access to the property `url`, which they can alter as much as they like to do whatever it is they need to do, with each new scenario getting a newly initalized world that is back to the default value +That gives all test steps access to the property `url`, which they can alter as much as they like to do whatever it is they need to do, with each new scenario getting a newly initialised world that is back to the default value ### Scoping @@ -136,7 +136,7 @@ end ### Trackable history -Finally, this is something we've had a few people had implemented independently and Atlas felt like the perfect place to formalise this functionality. There are times where a test might want to validate a sequence, such as the order pages have been visited in a journey through a website. Historically, people were creating various data structures to store these details, however in Atlas all properties that are initialised also come with a history field that stores an array of all previous values that that property has held. Take the following example: +Finally, this is something we've had a few people had implemented independently and Atlas felt like the perfect place to formalise this functionality. There are times where a test might want to validate a sequence, such as the order pages have been visited in a journey through a website. Historically, people were creating various data structures to store these details, however in Atlas all properties that are initialised also come with a history field that stores an array of all previous values that that property has held. Take the following example: ```ruby World do @@ -148,7 +148,7 @@ end ... Given 'the customer hits the submit button' do - artefacts.journey_status = fetch_journey_status # Should now be submitted + artefacts.journey_status = fetch_journey_status # Should now be submitted end ... @@ -179,4 +179,4 @@ define_singleton_method :"#{name}=" do |arg| end ``` -As you can see, we create the history as an empty array and define a getter for it. Then, as part of the definition of the setter for the actual property, we have line that pushes the value that is being overwritten into the history. This means that the setter can't be called without also updating the relevant history and ensures we are can access whatever previous values we require with no additional setup required when actually writing the tests. \ No newline at end of file +As you can see, we create the history as an empty array and define a getter for it. Then, as part of the definition of the setter for the actual property, we have line that pushes the value that is being overwritten into the history. This means that the setter can't be called without also updating the relevant history and ensures we are can access whatever previous values we require with no additional setup required when actually writing the tests. diff --git a/content/posts/2023-11-linux-timeout-command/index.md b/content/posts/2023-11-linux-timeout-command/index.md index 3a899eec..58d936ae 100644 --- a/content/posts/2023-11-linux-timeout-command/index.md +++ b/content/posts/2023-11-linux-timeout-command/index.md @@ -15,44 +15,50 @@ the AWS stacks within their limit. To ensure these steps run before the build times out, you can wrap your cucumber command in the linux timeout command within the drone pipeline -example: -```` -commands: - - cd functional-tests || exit 1 - - timeout -k 10 10m bundle exec rake test -```` +Example commands: + +```shell +cd functional-tests || exit 1 +timeout -k 10 10m bundle exec rake test +``` ## Linux Timeout + The timeout command is a command-line utility that will run the specified command for a given period of time, once that period of time is reached if the given command is still running it will be terminated. -timeout command: -```` +Timeout command: + +```shell timeout [OPTIONS] DURATION COMMAND [ARG]… -```` +``` ### Duration + The duration is an integer followed by one of the following units, if no unit is used then it defaults to seconds: -* s - seconds (default) -* m - minutes -* h - hours -* d - days +- s - seconds (default) +- m - minutes +- h - hours +- d - days ### Signal flag -If no signal is given, timeout sends the SIGTERM signal to the managed command when the time limit is reached. You can specify which signal to send + +If no signal is given, timeout sends the SIGTERM signal to the managed command when the time limit is reached. You can specify which signal to send using the -s (--signal) option. ### Kill-after flag -To ensure the command is killed, you can use the -k (--kill-after) flag with a specified time period. When the time limit is reached, the timeout command + +To ensure the command is killed, you can use the -k (--kill-after) flag with a specified time period. When the time limit is reached, the timeout command will then send the SIGKILL signal. ### Preserve-status flag -There is a preserved status flag that can be passed in, when this is used timeout will always returns 124 if it times out. +There is a preserved status flag that can be passed in, when this is used timeout will always returns 124 if it times out. ## Conclusion -Prior to introducing this command into our pipelines, we had many drone build failures where, for a variety or reasons, the functional test step would hang and the + +Prior to introducing this command into our pipelines, we had many drone build failures where, for a variety or reasons, the functional test step would hang and the build would time out, our 'AWS clean up step' would not execute and this resulted requiring manual intervention to clean them up (before we exceed our capacity). Since the introduction of this command we have seen a marked reduction in our drone builds timing out and not clearing down our AWS stacks. diff --git a/content/posts/2023-12-logging-blocks/index.md b/content/posts/2023-12-logging-blocks/index.md index 9568a18c..493d5e30 100644 --- a/content/posts/2023-12-logging-blocks/index.md +++ b/content/posts/2023-12-logging-blocks/index.md @@ -34,17 +34,17 @@ logger.warn('DB connection timed out. Reconnecting') [2023-12-15 09:09:49 45f91875] WARN -- : DB connection timed out. Reconnecting ``` -As you can see, we're only getting one log statement, with the `info` not being printed. But what does that mean under the bonnet? Let's take a look at what happens if we pass in a more complex parameter. +As you can see, we're only getting one log statement, with the `info` not being printed. But what does that mean under the bonnet? Let's take a look at what happens if we pass in a more complex parameter. ## Passing parameters -Consider the following code snippet: +Consider the following code snippet: ```ruby logger.info("Current active transactions: #{TransactionCounter.count_transactions}") ``` -Not much fancier than what we were doing previous. We're now just interpolating the response from the method `count_transactions` into the message we are logging out. We don't know what is in that method but it could be quite a costly call with lots of calculations going on behind it, so if we are in an environment where we aren't logging anything at `info` level we probably don't want to make that call. Now, suppose we add the following line somewhere in `count_transactions`: +Not much fancier than what we were doing previous. We're now just interpolating the response from the method `count_transactions` into the message we are logging out. We don't know what is in that method, but it could be quite a costly call with lots of calculations going on behind it, so if we are in an environment where we aren't logging anything at `info` level we probably don't want to make that call. Now, suppose we add the following line somewhere in `count_transactions`: ```ruby puts "Starting transaction count" @@ -69,9 +69,9 @@ logger.info { "Current active transactions: #{TransactionCounter.count_transacti Now, we're passing a block into the logger and the logger will, assuming it is logging out the current log level, log out the value returned by the block. The block above will return the same string as before, so let's take a look and see if anything is logged this time: ```sh - +# (no output) ``` That's more like it. This time we aren't calling `count_transactions`. Now, let's take a look at why that is. -Whereas before we were seeing the string fully interpolated before the `info` method was called, we're now offloading that into the block we're passing into the logger. As the logger is now in control of calling that block to get the value to log out, it defers doing so until it knows if it should be logging at the current level. Thus, for logs that are too low level, we don't make any expensive calls and save a few CPU cycles. \ No newline at end of file +Whereas before we were seeing the string fully interpolated before the `info` method was called, we're now offloading that into the block we're passing into the logger. As the logger is now in control of calling that block to get the value to log out, it defers doing so until it knows if it should be logging at the current level. Thus, for logs that are too low level, we don't make any expensive calls and save a few CPU cycles. diff --git a/content/posts/2023-12-spring-security-aws-kms/index.md b/content/posts/2023-12-spring-security-aws-kms/index.md index df1792ff..967c85a5 100644 --- a/content/posts/2023-12-spring-security-aws-kms/index.md +++ b/content/posts/2023-12-spring-security-aws-kms/index.md @@ -10,74 +10,76 @@ ShowToc: true TocOpen: true --- -As part of our proof of concepts in to the adoption of One Login we setup a Spring Security OAuth 2.0 +As part of our proof of concepts in to the adoption of One Login we setup a Spring Security OAuth 2.0 demo that tested out the [integration guide](https://docs.sign-in.service.gov.uk/integrate-with-integration-environment/integrate-with-code-flow/#make-a-token-request) provided by Government Digital Service (GDS). -Spring security has long had great OAuth2.0 support from both the server and client elements. Over the last year spring security +Spring security has long had great OAuth2.0 support from both the server and client elements. Over the last year spring security added support for the private_key_jwt client authentication method as part of the authorization code grant flow. [Spring Security GitHub ref](https://github.com/spring-projects/spring-security/pull/9933) -The configuration for the key signing requires both the public and private key to be available to the application. -Key Management Service (KMS) by Amazon Web Services (AWS) provides services to centrally manage your keys for encryption +The configuration for the key signing requires both the public and private key to be available to the application. +Key Management Service (KMS) by Amazon Web Services (AWS) provides services to centrally manage your keys for encryption and signing and is an option to allow a more centralised mechanism for key rotation and policy management. -In order to add this support in to Spring it requires a number of customisations to be made which will be explained in -this post. If your AuthZ server also supports elliptic-curve digital signature algorithms (ECDSA) for the ID token I will +In order to add this support in to Spring it requires a number of customisations to be made which will be explained in +this post. If your AuthZ server also supports elliptic-curve digital signature algorithms (ECDSA) for the ID token I will outline the additional Bean configuration required as Spring security defaults to RS256 supported with RSA keys. ## private_key_jwt -The private key jwt client authentication method requires a jwt token to be sent alongside client id, code as a client -assertion to the token endpoint. This jwt will need to be signed and then sent in a client_assertion parameter to the -auth server. A number of algorithms are supported by various auth servers and can be found by visiting the configuration + +The private key jwt client authentication method requires a jwt token to be sent alongside client id, code as a client +assertion to the token endpoint. This jwt will need to be signed and then sent in a client_assertion parameter to the +auth server. A number of algorithms are supported by various auth servers and can be found by visiting the configuration endpoint of the auth server. Full details of the spec can be found here [rfc 7523](https://www.rfc-editor.org/rfc/rfc7523.html) -The first configuration change required is to switch out the default access token response client as part of the -HttpSecurity config. This provides the ability to customise the token endpoint request parameters to enrich with the client +The first configuration change required is to switch out the default access token response client as part of the +HttpSecurity config. This provides the ability to customise the token endpoint request parameters to enrich with the client assertion signed by kms. ```java @Bean public Security FilterChain filterchain(HttpSecurity http) throws Exception { - .... + // ... http.oauth2Login(oauth2Login -> oauthLogin.tokenEndpoint(tokenEndpoint -> tokenEndpoint.accessTokenResponseClient(accessTokenResponseClient))). .oauth2Client.authorizationCodeGrant(authorizationCodeGrant -> authorizationCodeGrant.accessTokenResponseClient(accessTokenResponseClient))); - .... + // ... http.build(); } ``` ## OAuth2AccessTokenResponseClient -In order to override the request entity handling to add support for the kms signing you need to add a custom request + +In order to override the request entity handling to add support for the kms signing you need to add a custom request entity converter to the access token response client that you create. ```java -... +// ... @Bean public OAuth2AccessTokenResponseClient It is important to note whether or not you are creating the beans or are they being managed and created by Spring +> as you could incur a number of errors if the objects are manually instantiated. The converter is the main class that allows the overriding of the request parameters. ```java @Component public class CustomKMSJWTClientAuthNConverter implements Converter> { - private OAuth2AuthorizationCodeGrantRequestEntityConverter defaultConverter; public CustomKMSJWTClientAuthNConverter () { @@ -96,21 +98,21 @@ public class CustomKMSJWTClientAuthNConverter implements Converter idTokenDecoderFactory() { OidcTokenDecoderFactory idTokenDecoderFactory = new OidcTokenDecoderFactory(); idTokenDecoderFactory.setJwsAlgorithmResolver(clientRegistration -> SignatureAlgorithm.RS512); -return idTokenDecoderFactory; + return idTokenDecoderFactory; } -``` \ No newline at end of file +``` diff --git a/content/posts/2024-03-streamline-application-processing/images/ape-and-dape.svg b/content/posts/2024-03-streamline-application-processing/images/ape-and-dape.svg index 9f80f3c1..ba6a893c 100644 --- a/content/posts/2024-03-streamline-application-processing/images/ape-and-dape.svg +++ b/content/posts/2024-03-streamline-application-processing/images/ape-and-dape.svg @@ -1,3 +1,202 @@ - - -
AWS Step Functions workflow
Start Processintg
Received
Reception
Hook
SettlePayment
Preparation
Hook
AWS Step Functions workflow
Reception
Preparation
Generic Application Processing
Drivers Application Processing
Submit
Hook
Submit
 \ No newline at end of file + + + + + + + + + +
+
+
+ AWS Step Functions workflow +
+
+
+ + + + + +
+
+
+ Start Processintg +
+
+
+ + + + + + +
+
+
+ Received +
+
+
+ + + + + + +
+
+
+ Reception +
+ Hook +
+
+
+ + + + + + +
+
+
+ SettlePayment +
+
+
+ + + + + + +
+
+
+ Preparation +
+ Hook +
+
+
+ + + + + + + + + + + + + + +
+
+
+ AWS Step Functions workflow +
+
+
+ + + + + + +
+
+
+ Reception +
+
+
+ + + + + + +
+
+
+ Preparation +
+
+
+ + + + + + + + + + + + + +
+
+
+ Generic Application Processing +
+
+
+ + + + + +
+
+
+ Drivers Application Processing +
+
+
+ + + + + + +
+
+
+ Submit +
+ Hook +
+
+
+ + + + + + + + +
+
+
+ Submit +
+
+
+ + + + + + + diff --git a/content/posts/2024-03-streamline-application-processing/images/application-process-engine.jpg b/content/posts/2024-03-streamline-application-processing/images/application-process-engine.jpg new file mode 100644 index 00000000..6301cce1 Binary files /dev/null and b/content/posts/2024-03-streamline-application-processing/images/application-process-engine.jpg differ diff --git a/content/posts/2024-03-streamline-application-processing/images/application-process-engine.png b/content/posts/2024-03-streamline-application-processing/images/application-process-engine.png deleted file mode 100644 index c33a9132..00000000 Binary files a/content/posts/2024-03-streamline-application-processing/images/application-process-engine.png and /dev/null differ diff --git a/content/posts/2024-03-streamline-application-processing/index.md b/content/posts/2024-03-streamline-application-processing/index.md index cb6aaabf..0c6cdadc 100644 --- a/content/posts/2024-03-streamline-application-processing/index.md +++ b/content/posts/2024-03-streamline-application-processing/index.md @@ -11,64 +11,70 @@ TocOpen: true --- ## Introduction + Harnessing the power of AWS Step Functions, the Driver and Vehicle Licensing Agency (DVLA) embarks on a transformative journey to streamline its operations. With over 50 million registered drivers and 40 million vehicles, the DVLA navigates various applications, from driving licences to vehicle registrations. This blog unveils the approach adopted by DVLA to transform its application process, ensuring flexibility and customisation for each application type. ## Background + As the custodian of millions of driver and vehicle records, DVLA grapples with the complexity of managing diverse applications. From initial licence acquisition to periodic photo licence renewals and vehicle-related transactions, the spectrum of services demands a cohesive yet adaptable framework. ## Challenge + DVLA’s organisational structure traditionally compartmentalises teams into the driver or vehicle-focused domains. Integrating disparate knowledge bases becomes an arduous task, compounded by the intricate regulations governing both drivers and vehicles. ## Solution approach -DVLA carried out the analysis of its application landscape, identifying common phases traversed by all applications. This critical insight formed the foundation for crafting a unified application workflow, capable of seamlessly orchestrating diverse applications through their respective journeys. +DVLA carried out the analysis of its application landscape, identifying common phases traversed by all applications. This critical insight formed the foundation for crafting a unified application workflow, capable of seamlessly orchestrating diverse applications through their respective journeys. ## Technology choice + Building upon existing expertise with AWS infrastructure, DVLA leveraged AWS Step Functions to architect its solution. The decision was reinforced by the robustness and adaptability offered by AWS Step Functions, aligning seamlessly with DVLA’s cloud-native strategy. ## Introducing the application process engine aka APE + This is where the application process engine was born, a dynamic workflow engine designed to navigate applications through various stages. APE embodies versatility, executing generic business functionalities common to most applications. -{{
}} +{{
}} -The aforementioned diagram offers a birdseye view of the orchestration of the application across its diverse phases. The workflows dedicated to driver applications are compartmentalised and self-contained, executing only the driving licence application-specific steps essential for completing each phase. +The diagram offers a birdseye view of the orchestration of the application across its diverse phases. The workflows dedicated to driver applications are compartmentalised and self-contained, executing only the driving licence application-specific steps essential for completing each phase. ## Workflow within a workflow 😕? + In our approach to handling the workflow for driving licences and vehicle-related logic, we opted to define sub-workflows for each application phase. For instance, in the scenario where licence holders are mandated to renew their photo on their driving licence, they now have the option to upload a photo of their choice, presenting a unique "selfie opportunity." Should they choose this route, a designated clerk within the DVLA is tasked with conducting meticulous checks on the submitted image to ensure compliance with standards. However, the preparatory steps involved in photo submission and the subsequent review by a clerk hold no relevance in the context of vehicle applications or instances where individuals seek to update their address information on their driving licence or vehicle - wherein no photo submission is required. - -{{
}} +{{
}} ## Harnessing AWS Step Functions direct integrations + Within the extensive repertoire of direct integrations offered by AWS Step Functions, lies a notable inclusion: AWS Step Functions themselves (yes, start a workflow within a workflow). Leveraging this optimized integration, we seamlessly triggered sub-workflows tailored to the specific phases of driver or vehicle applications directly from our application process engine. This innovative approach draws inspiration from the [ServerlessLand](https://serverlessland.com/workflows/workflow-within-a-workflow-cdk) website, underscoring our commitment to efficiency and best practices in workflow architecture. AWS simplifies the process of defining steps necessary to trigger sub-workflows. We leverage the [Serverless Framework](https://www.serverless.com/framework/docs) for deploying our applications onto AWS which further enhances this seamless integration. Below is an example of how a sub-workflow is triggered: ```yaml - CheckReceptionHook: - Type: Choice - Choices: - - Variable: $.hooks.receptionStateMachine - "IsPresent": true - Next: ReceptionHook - Default: SettlePayment - - ReceptionHook: - Type: Task - Resource: arn:aws:states:::states:startExecution.sync:2 - Parameters: - StateMachineArn.$: $.hooks.receptionStateMachine.arn - Input: - applicationId.$: $.applicationId - iteration.$: $.iteration - AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$: $$.Execution.Id - Name.$: States.Format('{}-{}-{}', $.hooks.receptionStateMachine.name, $.applicationId, $.iteration) - Retry: - # Some retry definitions - ResultPath: null - Next: SettlePayment +CheckReceptionHook: + Type: Choice + Choices: + - Variable: $.hooks.receptionStateMachine + "IsPresent": true + Next: ReceptionHook + Default: SettlePayment + +ReceptionHook: + Type: Task + Resource: arn:aws:states:::states:startExecution.sync:2 + Parameters: + StateMachineArn.$: $.hooks.receptionStateMachine.arn + Input: + applicationId.$: $.applicationId + iteration.$: $.iteration + AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$: $$.Execution.Id + Name.$: States.Format('{}-{}-{}', $.hooks.receptionStateMachine.name, $.applicationId, $.iteration) + Retry: + # Some retry definitions + ResultPath: null + Next: SettlePayment ``` Before delving into the defined steps outlined above, it's crucial to discuss certain preparatory steps. At the outset of processing, the application process engine initiates a pivotal step. Here, logic is deployed to ascertain the application type, delineate the required sub-workflows, and furnish the necessary Amazon Resource Names (ARNs) for their execution. @@ -82,25 +88,33 @@ This orchestration empowers the sub-workflows to seamlessly adapt to the idiosyn ### Advantages of this approach #### Addressing the challenge + One of the primary hurdles we encounter revolves around the domain expertise of our teams, particularly concerning drivers or vehicles. This approach empowers driver and vehicle teams to craft their specialised sub-workflows tailored to their applications. These teams seamlessly integrate these sub-workflows into the application process engine, thereby effectively fulfilling the application. #### Streamlined testing + Testing becomes notably simplified with this approach, as clear boundaries are delineated for testing purposes. The application process engine operates as an independent deployment, facilitating comprehensive testing within a controlled environment. Furthermore, the project can undergo testing in isolation. Simple stubs can be readily implemented for the sub-workflows, facilitating smooth progression through various phases. Each individual sub-workflow operates as a self-contained entity, enabling isolated triggering and testing. #### Efficient utilisation of common functionality + Every application inherits the shared functionalities embedded within the application process engine, obviating the need for duplicated steps. This streamlined approach empowers teams to focus their efforts on refining their unique business logic and workflow, thereby enhancing overall efficiency and productivity. ### Navigating challenges in implementation + #### Designing a robust generic application process engine + Despite our analysis of each application type within the DVLA, designing a comprehensive application process engine presents formidable challenges. Anticipating the impact of our design decisions across diverse application types proves inherently complex. To mitigate this challenge, communication and collaboration are paramount. Ensuring that all relevant stakeholders are engaged and have oversight of the changes helps minimise the risk of unintended consequences. #### Deploying to an Active Production Environment + Deploying application workflows to a live production environment introduces another layer of complexity. As these workflows actively process applications, any alterations to the underlying sub-workflow state must be executed with utmost caution. Deploying changes while applications are in progress runs the risk of inconsistencies or disruptions in workflow execution. Hence, we attempt to incorporate robust backward compatibility and versioning mechanisms to navigate the evolving landscape effectively. #### End-to-end testing dilemmas + While the benefits of testing in isolation are undeniable, they also introduce challenges, particularly in the realm of integration testing. Integrating isolated components developed by different teams necessitates rigorous integration testing to ensure seamless interoperability. Effective communication becomes paramount, especially when teams may naturally prioritise their component's delivery over integration testing. At DVLA, we mitigate this challenge by dedicating a specialised team solely focused on integration testing, ensuring comprehensive validation in a fully integrated environment with an independent view and approach. -## Conclusion +## Conclusion + In summary, the DVLA's adoption of AWS Step Functions and the development of the application process engine, represent significant strides towards modernising its application processing. We are thrilled to announce that we have successfully gone live with the ten-year licence renewal process, marking a significant milestone in our digital transformation journey. -Time will tell whether the design decisions result in an overall improvement in migrating and delivering applications onto the new platform. \ No newline at end of file +Time will tell whether the design decisions result in an overall improvement in migrating and delivering applications onto the new platform. diff --git a/content/posts/2024-07-adding-numerous-tags-to-paralleltests-in-cucumber-elegantly/index.md b/content/posts/2024-07-adding-numerous-tags-to-paralleltests-in-cucumber-elegantly/index.md index c472b6ea..f5191765 100644 --- a/content/posts/2024-07-adding-numerous-tags-to-paralleltests-in-cucumber-elegantly/index.md +++ b/content/posts/2024-07-adding-numerous-tags-to-paralleltests-in-cucumber-elegantly/index.md @@ -4,8 +4,7 @@ title: "TiL: Adding numerous tags to ParallelTests in cucumber elegantly " description: "Creating numerous tags to be included/excluded for ParallelTests in cucumber elegantly" draft: false date: 2024-07-02 -tags: - ["Ruby", "ParallelTests", "Tags", "Cucumber", "Testing", "Today I Learned"] +tags: ["Ruby", "ParallelTests", "Tags", "Cucumber", "Testing", "Today I Learned"] categories: ["TIL", "Ruby", "Cucumber", "ParallelTests", "Testing"] ShowToc: true TocOpen: true diff --git a/content/posts/2024-07-non-blocking-ruby-methods/index.md b/content/posts/2024-07-non-blocking-ruby-methods/index.md index d9d174cb..1982c9e1 100644 --- a/content/posts/2024-07-non-blocking-ruby-methods/index.md +++ b/content/posts/2024-07-non-blocking-ruby-methods/index.md @@ -15,12 +15,15 @@ There are lot of options for concurrency in Ruby, from spawning child processes In my use-case, I want to do some work and, in the background, perform some network requests. Network I/O is blocking, so I want to make sure I'm doing productive work on the main thread while I wait for the network requests to complete. ## Fibers + Fibers are lightweight but fairly low-level primitives. They allow you to pause and resume execution at will. Whenever they pause, they yield control back to the caller, which can then resume the Fiber at a later time. This is great for enumerators which can process a loop and return the computed value to the caller. In my case I want to yield control whenever an operation blocks. Fortunately, Ruby has introduced a Fiber scheduler interface and the `async` gem provides a high-level API which handles the gnarly details of just this use-case. ## Example + With this in my Gemfile: + ```ruby gem "async", "~> 2.14" ``` @@ -48,16 +51,16 @@ A call to `API.perfom` will create the fiber, register it with the scheduler and Let's put it to use: - ```ruby - require 'async/scheduler' - require 'httparty' - require_relative 'api' +```ruby +require 'async/scheduler' +require 'httparty' +require_relative 'api' - puts 'Starting up...' - 5.times do |i| - API.perform(i) - end - puts 'Finished!' +puts 'Starting up...' +5.times do |i| + API.perform(i) +end +puts 'Finished!' ``` The code will schedule 5 fibers to run concurrently, each making a network request. The main thread will continue to run and print 'Finished!' before the network requests complete. @@ -80,27 +83,28 @@ end run App.freeze.app ``` -When I run the the client code, I see the following output: +When I run the client code, I see the following output: ```shell Starting up... -Making request...0 -Making request...1 -Making request...2 -Making request...3 -Making request...4 + Making request...0 + Making request...1 + Making request...2 + Making request...3 + Making request...4 Finished! - Done with request 3: Hello world! - Done with request 4: Hello world! - Done with request 1: Hello world! - Done with request 2: Hello world! - Done with request 0: Hello world! + Done with request 3: Hello world! + Done with request 4: Hello world! + Done with request 1: Hello world! + Done with request 2: Hello world! + Done with request 0: Hello world! ``` The main thread continues to run while the network requests are in progress. When the requests complete, the fibers are resumed and the results are printed to the console. All the fibers run to completion before the program exits. Nice. ## Bonus section -Roda is pretty speedy framework but I'm running here with the basic Rackup server. I can queue about 500 Fibers before saturating the server and it starts rejecting connections. The Falcon web server is written by Samuel Williams, the same author as the Async gem, and is designed to work well with Async. It's a great choice for running high-performance Ruby web servers, so let's try that. + +Roda is pretty speedy framework, but I'm running here with the basic Rackup server. I can queue about 500 Fibers before saturating the server, and it starts rejecting connections. The Falcon web server is written by Samuel Williams, the same author as the Async gem, and is designed to work well with Async. It's a great choice for running high-performance Ruby web servers, so let's try that. To use Falcon, add it to your server's Gemfile: diff --git a/content/posts/2024-07-ruby-gsub-hashes/index.md b/content/posts/2024-07-ruby-gsub-hashes/index.md index 99cb813c..5c3936f1 100644 --- a/content/posts/2024-07-ruby-gsub-hashes/index.md +++ b/content/posts/2024-07-ruby-gsub-hashes/index.md @@ -24,15 +24,15 @@ But what if you wanted to do something different for each match? Well, because ` # => "There are several cakes and a few teapots on the table" ``` -This can quickly become unmanagable. However, there is another way we can interact with `gsub`. Instead of simply giving it a string as the second argument, we can pass in a hash of matches, with the keys being potential matches and the values being the replacement strings. +This can quickly become unmanageable. However, there is another way we can interact with `gsub`. Instead of simply giving it a string as the second argument, we can pass in a hash of matches, with the keys being potential matches and the values being the replacement strings. ```ruby -tea_party_matches ={ - '3' => 'a few', - '5' => 'several' +tea_party_matches = { + '3' => 'a few', + '5' => 'several' } 'There are 5 cakes and 3 teapots on the table'.gsub(/\d/, tea_party_matches) # => There are several cakes and a few teapots on the table ``` -This tends to be much more readable, particularly when you get into more real-world examples with complex regex statements matching several different things you might want to replace. \ No newline at end of file +This tends to be much more readable, particularly when you get into more real-world examples with complex regex statements matching several different things you might want to replace. diff --git a/content/posts/2024-09-ruby-exec-vs-system/index.md b/content/posts/2024-09-ruby-exec-vs-system/index.md index e4955bba..4b31f354 100644 --- a/content/posts/2024-09-ruby-exec-vs-system/index.md +++ b/content/posts/2024-09-ruby-exec-vs-system/index.md @@ -4,21 +4,21 @@ title: "TiL: The difference between exec and system in Ruby" description: "What exactly are exec and system doing and why do they differ?" draft: false date: 2024-09-23 -tags: ["Ruby", "exec", 'system', "Today I Learned"] +tags: ["Ruby", "exec", "system", "Today I Learned"] categories: ["TIL", "Ruby"] ShowToc: true TocOpen: true --- -Ruby provides a few different ways to execute commands against the underlying kernel programmatically, but they all work slightly differently. The simplest way to execute a command againts the shell is to surround it in backticks. For example, the following will run the command `date` and return whatever was output to `$stdout`. +Ruby provides a few different ways to execute commands against the underlying kernel programmatically, but they all work slightly differently. The simplest way to execute a command against the shell is to surround it in backticks. For example, the following will run the command `date` and return whatever was output to `$stdout`. ```ruby current_date = `date` ``` -However, there are also more explict methods you can call. In particular, there is `exec` and `system` which look like they do the same thing at a quick glance. However, they actually work very differently once you get down into what they are doing. +However, there are also more explicit methods you can call. In particular, there is `exec` and `system` which look like they do the same thing at a quick glance. However, they actually work very differently once you get down into what they are doing. -# Exec +## Exec `exec` works by replacing the process it is called within with the one that is passed into it. This can either be a command to be executed or the path to an executable, but either way the current process will be replaced with what is passed in. This means that when the newly created process exits, it won't have a ruby application to return to. Take the following code: @@ -36,7 +36,7 @@ Getting the date Mon Sep 23 10:22:14 BST 2024 ``` -# System +## System The basic interface with `system` is very similar. When provided with either a command as a string or the path to an executable, it will be started in a process. The difference is that, rather than replacing the current process, it will instead create a child process. When this process exits it will return back to the ruby process and continue executing, returning `true` if the command exits with status 0 and `false` if it exits with a non-zero integer. Let us return to the previous example: @@ -46,6 +46,7 @@ puts 'Getting the date' system 'date' puts value * 2 ``` + Now that we using system, the final line will execute and our output will look like the following: ```shell @@ -56,6 +57,6 @@ Mon Sep 23 10:22:19 BST 2024 Much more useful if you need to quickly run a command as part of your code and don't wish to fully cede control to a different program. -# A word of warning +## A word of warning -As with anything like this, it is always worth keeping in mind the risk of executing a command aganst the shell directly. No matter which way you are doing it, never execute anything you aren't confident you understand everything it can do. \ No newline at end of file +As with anything like this, it is always worth keeping in mind the risk of executing a command against the shell directly. No matter which way you are doing it, never execute anything you aren't confident you understand everything it can do. diff --git a/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets banner screenshot.jpg b/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets banner screenshot.jpg new file mode 100644 index 00000000..7a58fbb2 Binary files /dev/null and b/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets banner screenshot.jpg differ diff --git a/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets banner screenshot.png b/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets banner screenshot.png deleted file mode 100644 index 567af24e..00000000 Binary files a/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets banner screenshot.png and /dev/null differ diff --git a/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets pop up box.jpg b/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets pop up box.jpg new file mode 100644 index 00000000..4cbd88b5 Binary files /dev/null and b/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets pop up box.jpg differ diff --git a/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets pop up box.png b/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets pop up box.png deleted file mode 100644 index 359fa3a7..00000000 Binary files a/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/images/Drone secrets pop up box.png and /dev/null differ diff --git a/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/index.md b/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/index.md index e325e598..dadbe4c6 100644 --- a/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/index.md +++ b/content/posts/2024-10-toggle-ruby-tests-through-ci-secrets/index.md @@ -10,10 +10,8 @@ ShowToc: true TocOpen: true --- +## _TiL_ how to toggle Ruby scenarios to be run using CI pipeline secrets -# *TiL* **how to toggle Ruby scenarios to be run using CI pipeline secrets** - ---- ## Scenario In a relatively niche scenario, we have functionality that is being temporarily removed while new code is being developed. @@ -21,6 +19,7 @@ In a relatively niche scenario, we have functionality that is being temporarily While we still want to keep our current scenarios for function X to use them again when it's reintroduced, if they are left to run in our pipeline, every build will fail until it is reintroduced again. --- + ### How do we get around this? Well to start off with, because we're good testers, we need to write new scenarios that would check to see that the expected errors will be returned once the code is removed, providing us with confidence that it'll be working correctly. @@ -33,7 +32,7 @@ We can use CI pipeline secrets to our advantage, in our pipeline config file we' ```yaml RUN_FUNCTION_X: - from_secret: RUN_FUNCTION_X # Either ':yes' or ':no' -- set as secret + from_secret: RUN_FUNCTION_X # Either ':yes' or ':no' -- set as secret ``` Best practice would be to set this variable with a default value. @@ -45,7 +44,6 @@ RUN_FUNCTION_X: <%= ENV['RUN_FUNCTION_X'] || :yes > And then in our env.rb, we will set a boolean value to `RUN_FUNCTION_X` by checking if the drone secret has been added. - ```ruby # Switch the drone secret from :yes to :no to toggle test functionality RUN_FUNCTION_X = SimpleSymbolize.symbolize(Settings.RUN_FUNCTION_X) == :yes @@ -64,19 +62,18 @@ Using conditionals, `if` and `unless`, we can use our hooks to skip one set of t Before('@UNHAPPY_PATH_FUNCTION_X') do if RUN_FUNCTION_X Logger.info { 'Skip function X scenarios are set to be ignored. Skipping scenario.' } - raise Core::Test::Result::Skipped, 'Scenario Skipped' + raise Core::Test::Result::Skipped, 'Scenario Skipped' end end Before('@FUNCTION_X') do unless RUN_FUNCTION_X Logger.info { 'Function X scenarios are set to be ignored. Skipping scenario.' } - raise Core::Test::Result::Skipped, 'Scenario Skipped' + raise Core::Test::Result::Skipped, 'Scenario Skipped' end end ``` - ### The Tests We need to make it clear which test scenarios are going to be skipped, and which tests will be our unhappy path tests, so we add our custom Cucumber tags to our feature files. @@ -94,27 +91,23 @@ Given a customer logs into the system When the customer applies for function x Then the system produces the following error: | status | code | detail | -| 400 | 123 | Function X error | -```` +| 400 | 123 | Function X error | +``` ### And Finally -{{
}} +{{
}} With all this in place, as soon as we require function X scenarios to stop running and our unhappy path tests to start, all we need to do is add a secret to our CI pipeline called `RUN_FUNCTION_X` and set it to `:no`. -{{
}} +{{
}} Then all you have to do is delete the secret once the function is reintroduced, the removal of the config, unhappy path tests and tags afterwards is up to your discretion depending on whether that functionality is to be temporarily removed again in the future or not. - ## TL;DR If a function is temporarily removed, we can skip our tests for that functionality and run tests to ensure that the unavailable function returns the correct error. -Setting CI pipeline secret that's linked to an environment variable in our CI pipeline config file and in a config file, best set to a default of truthy or true value, which is set to a boolean value by the env.rb before each run. +Setting CI pipeline secret that's linked to an environment variable in our CI pipeline config file and in a config file, best set to a default of truthy or true value, which is set to a boolean value by the env.rb before each run. Using that boolean, we decide whether to skip the function X scenarios and run our unhappy path tests using before hooks linked to Cucumber tags. - - ---- \ No newline at end of file diff --git a/content/posts/2025-02-postman-paf/index.md b/content/posts/2025-02-postman-paf/index.md index 7036c781..693b1f4f 100644 --- a/content/posts/2025-02-postman-paf/index.md +++ b/content/posts/2025-02-postman-paf/index.md @@ -16,50 +16,49 @@ We've recently released a library called [postman-paf-js](https://github.com/dvl ## Why is Postman Paf required? -To put it simply, UK addresses are more complicated than they might seem. But aren't addresses just a few set lines that appear on your letters? Unfortunately, it's not that simple! Most addresses in the UK are stored in a database called the Postal Address File (PAF), which is owned and managed by the -Royal Mail. This database includes over 30 million UK postal delivery addresses and 1.8 million postcodes. However, these addresses are not stored in the familiar format you -see on your envelope. Instead, they are stored in a specific data structure, as shown -[here](https://www.poweredbypaf.com/wp-content/uploads/2017/07/Latest-Programmers_guide_Edition-7-Version-6-1.pdf#page=12) - +To put it simply, UK addresses are more complicated than they might seem. But aren't addresses just a few set lines that appear on your letters? Unfortunately, it's not that simple! Most addresses in the UK are stored in a database called the Postal Address File (PAF), which is owned and managed by the +Royal Mail. This database includes over 30 million UK postal delivery addresses and 1.8 million postcodes. However, these addresses are not stored in the familiar format you +see on your envelope. Instead, they are [stored in a specific data structure, as shown here](https://www.poweredbypaf.com/wp-content/uploads/2017/07/Latest-Programmers_guide_Edition-7-Version-6-1.pdf#page=12) Example of a PAF address - ```JSON + +```json { - "thoroughfareName": "NORTH EAST SECTOR", - "doubleDependentLocality": "BOURNEMOUTH INTERNATIONAL AIRPORT", - "postcode": "BH23 6NE", - "dependentLocality": "HURN", - "postTown": "CHRISTCHURCH", - "buildingName": "A B P 21", - "organisationName": "BREATHE SAFETY LTD" + "thoroughfareName": "NORTH EAST SECTOR", + "doubleDependentLocality": "BOURNEMOUTH INTERNATIONAL AIRPORT", + "postcode": "BH23 6NE", + "dependentLocality": "HURN", + "postTown": "CHRISTCHURCH", + "buildingName": "A B P 21", + "organisationName": "BREATHE SAFETY LTD" } ``` This data is structured very differently from the address format that appears on your envelope. Before these PAF addresses (which we at the DVLA call "Structured Addresses") can be used for postal mail, they must be converted into a format recognized by the Royal Mail. We refer to addresses in this format as "Unstructured Addresses." - ## Rules are rules To convert addresses from a Structured format to an Unstructured format, certain rules must be applied to each element of the address to ensure it is ordered correctly. -The Royal Mail has provided a developer guide detailing how to correctly perform this conversion, which includes several rules and exceptions that need to be followed +The Royal Mail has provided a developer guide detailing how to correctly perform this conversion, which includes several rules and exceptions that need to be followed in the correct order. -There are 4 ordering notes, 7 rules, and 4 exceptions that must be applied to a PAF address, all in the correct order, to ensure it is converted according to the Royal +There are 4 ordering notes, 7 rules, and 4 exceptions that must be applied to a PAF address, all in the correct order, to ensure it is converted according to the Royal Mail's guidelines. ## Conversion example step by step An example of the process followed on converting a Structured address to an unstructured address can be found below: - ```JSON + +```json { - "structuredAddress": { - "buildingName": "ENDEAVOUR QUAY", - "subBuildingName": "THE DESIGN OFFICE", - "thoroughfareName": "MUMBY ROAD", - "postcode": "PO12 1AH", - "postTown": "GOSPORT" - } + "structuredAddress": { + "buildingName": "ENDEAVOUR QUAY", + "subBuildingName": "THE DESIGN OFFICE", + "thoroughfareName": "MUMBY ROAD", + "postcode": "PO12 1AH", + "postTown": "GOSPORT" + } } ``` @@ -74,37 +73,36 @@ line if there is no Thoroughfare information. In this case none of these exceptions apply. Otherwise, the Sub Building Name should appear on a line preceding the Building Name, Thoroughfare and Locality information. - - ```JSON + +```json { - "line1": "THE DESIGN OFFICE", - "line2": "ENDEAVOUR QUAY" + "line1": "THE DESIGN OFFICE", + "line2": "ENDEAVOUR QUAY" } ``` ### Step 2 - Order Thoroughfare/locality -```JSON +```json { - "line1": "THE DESIGN OFFICE", - "line2": "ENDEAVOUR QUAY", - "line3": "MUMBY ROAD" + "line1": "THE DESIGN OFFICE", + "line2": "ENDEAVOUR QUAY", + "line3": "MUMBY ROAD" } ``` -### Step 3 - Add in post town and postcode +### Step 3 - Add in post-town and postcode -```JSON +```json { - "line1": "THE DESIGN OFFICE", - "line2": "ENDEAVOUR QUAY", - "line3": "MUMBY ROAD", - "line5": "GOSPORT", - "postcode": "PO12 1AH" + "line1": "THE DESIGN OFFICE", + "line2": "ENDEAVOUR QUAY", + "line3": "MUMBY ROAD", + "line5": "GOSPORT", + "postcode": "PO12 1AH" } ``` - ## How we did it Originally, the address converter library was written in Java and forked from an existing library: [paf-address-format](https://github.com/steinfletcher/paf-address-format). However, this library was relatively old (last updated 5 years ago) and did not conform to all the up-to-date rules and exceptions outlined in the latest Royal Mail Developers Guide. @@ -120,26 +118,24 @@ We currently use the library within our internal Addressing services where it is The library is simple to use, the convertStructuredToUnstructured function needs to be imported from the library then simply pass in a structured address. ```javascript -const { convertStructuredToUnstructured } = require('postman-paf-js'); +const { convertStructuredToUnstructured } = require("postman-paf-js"); const structuredAddressToConvert = { - buildingNumber: '1', - thoroughfareName: 'Example Street', - postTown: 'City', - postcode: 'AA1 1AA', + buildingNumber: "1", + thoroughfareName: "Example Street", + postTown: "City", + postcode: "AA1 1AA" }; const convertedUnstructuredAddress = convertStructuredToUnstructured(structuredAddressToConvert); - ``` -The function will then return an unstructured (Address with up to 5 lines and a postcode ) in the royal mail format +The function will then return an unstructured (Address with up to 5 lines and a postcode) in the Royal Mail format ```javascript convertedUnstructuredAddress = { - line1: '1 Example Street', - line5: 'City', - postcode: 'AA1 1AA', + line1: "1 Example Street", + line5: "City", + postcode: "AA1 1AA" }; ``` - diff --git a/content/posts/2025-04-04-experimenting-with-rag-llm/index.md b/content/posts/2025-04-04-experimenting-with-rag-llm/index.md index 25ea6b4b..2a062bc4 100644 --- a/content/posts/2025-04-04-experimenting-with-rag-llm/index.md +++ b/content/posts/2025-04-04-experimenting-with-rag-llm/index.md @@ -10,9 +10,9 @@ ShowToc: true TocOpen: true --- -*Note well* Please ensure you consider and adhere to any policies and restrictions your organisation places on the use of data with AI and the selection of AI models. +_Note well_ Please ensure you consider and adhere to any policies and restrictions your organisation places on the use of data with AI and the selection of AI models. -I want to be able to ask an generative AI some questions while giving it the context from which I'd like it to use it's smarts to derive an answer. This is Retrieval-Augmented Generation (RAG). +I want to be able to ask a generative AI some questions while giving it the context from which I'd like it to use it's smarts to derive an answer. This is Retrieval-Augmented Generation (RAG). Being a Ruby engineer, I'm going to pick up my shiny red hammer to attack this problem. @@ -28,11 +28,11 @@ Because I'm super lazy and want to experiment by hand, I'm using the open source ## Chatting to the LLM -At the begining of this experiment, I wasn't sure which model I wanted to use or how to interface with it so I used the [`langchainrb`](https://github.com/patterns-ai-core/langchainrb) library which provides a high-level, pluggable interface. +At the beginning of this experiment, I wasn't sure which model I wanted to use or how to interface with it, so I used the [`langchainrb`](https://github.com/patterns-ai-core/langchainrb) library which provides a high-level, pluggable interface. I do need to install the [Open AI](https://rubygems.org/gems/ruby-openai) as well. -Then I'm going to configure the Open AI library to use my local server rather than the internet. When I create a LLM client, I need to tell it which model I'm going to use. Since Jan can only load one model, I'm going to use the same one for all interations. +Then I'm going to configure the Open AI library to use my local server rather than the internet. When I create a LLM client, I need to tell it which model I'm going to use. Since Jan can only load one model, I'm going to use the same one for all interactions. ```ruby require 'langchain' @@ -47,14 +47,17 @@ OpenAI.configure do |c| c.uri_base = 'http://127.0.0.1:1337/v1' end -llm = Langchain::LLM::OpenAI.new(api_key: 'locally-model-no-api-key', - default_options:{ - chat_model: MODEL, - completion_model: MODEL, - embedding_model: MODEL } ) +llm = Langchain::LLM::OpenAI.new( + api_key: 'locally-model-no-api-key', + default_options:{ + chat_model: MODEL, + completion_model: MODEL, + embedding_model: MODEL, + }, +) ``` -I also want an assistant client. An assistant stores context to make conversational interations more natural. I'm going to pass a block into the constructor which will be called as the response is streamed rather than wait until a complete result is received because I just want to print the response to the console as it is generated. +I also want an assistant client. An assistant stores context to make conversational interactions more natural. I'm going to pass a block into the constructor which will be called as the response is streamed rather than wait until a complete result is received because I just want to print the response to the console as it is generated. ```Ruby assistant = Langchain::Assistant.new( @@ -67,7 +70,6 @@ assistant = Langchain::Assistant.new( end ``` - ## Collecting my own data I need to turn my unstructured source information into something a machine can deal with. For this experiment, I actually used some of our HR policies: they're wordy, somewhat complex and the documents can be easily converted from Word to Markdown. I'm going to use Markdown section headers to identify coherent sections of text, something which works for these documents. @@ -139,21 +141,22 @@ loop do embeddings << llm.embed(text: query, model: 'llama3.2-3b-instruct').embedding context = db.entities.hybrid_search( - collection_name: 'hr', - search: embeddings.map { - { anns_field: 'vector', - data: [it], # Ruby v3.4 `it` block keyword - output_fields: ['text'], - limit: 5 - } }, - rerank: { - strategy: 'rrf', - params: { k: 10 } - }, - limit: 5, - output_fields: ['text'])['data'].map { - it['text'] - }.join("\n\n") + collection_name: 'hr', + search: embeddings.map { + { + anns_field: 'vector', + data: [it], # Ruby v3.4 `it` block keyword + output_fields: ['text'], + limit: 5, + } + }, + rerank: { + strategy: 'rrf', + params: { k: 10 } + }, + limit: 5, + output_fields: ['text'], + )['data'].map { it['text'] }.join("\n\n") prompt = <<~PROMPT Use the following pieces of information enclosed in tags and from previous contexts to provide an answer to the question enclosed in tags. Do not mention the tags in your answer. diff --git a/content/posts/2025-04-24-kaping-gem-project/index.md b/content/posts/2025-04-24-kaping-gem-project/index.md index 7796f36d..8520c44b 100644 --- a/content/posts/2025-04-24-kaping-gem-project/index.md +++ b/content/posts/2025-04-24-kaping-gem-project/index.md @@ -10,38 +10,38 @@ ShowToc: true TocOpen: true --- -***You can only stretch elastic so far before it goes KA-PING!*** +## You can only stretch elastic so far before it goes "KA-PING!" -The starting point for creating a new DVLA gem to integrate with AWS OpenSearch and ElasticSearch was the amount of documentation +The starting point for creating a new DVLA gem to integrate with AWS OpenSearch and ElasticSearch was the amount of documentation there is surrounding the technology. There is a lot for a reason, it's a very powerful and useful tool, -but the flip side is the learning curve involved in understanding all the features and query structure. +but the flip side is the learning curve involved in understanding all the features and query structure. Our use case is we wanted squads to have a simple way to retrieve specific driver test data to use in their tests without having to fully understand Elasticsearch and all its capabilities. -A lot of our tests require very specific data requirements which can result in complex search queries with multiple parameters and +A lot of our tests require very specific data requirements which can result in complex search queries with multiple parameters and search terms to consider. The code to write these complex queries can ballon into a very deep nested JSON. -Out test platform architecture is based on Ruby, so we wanted a Ruby way to write large queries easily, which lead to the +Out test platform architecture is based on Ruby, so we wanted a Ruby way to write large queries easily, which lead to the development of a query builder using the dot notation concept to chain the queries terms together. We didn't need every aspect of the full OpenSearch capabilities to begin with, so we started with a sub set of search terms which we commonly use. With this in mind there is further development work to cover more of the OpensSearch functions down the line. +## The Ka-Ping Ruby Gem -## The Ka-Ping Ruby Gem - -The Ka-Ping Ruby Gem enables the user to build complex ElasticSearch DSL Queries for searching and filtering large data sets without +The Ka-Ping Ruby Gem enables the user to build complex ElasticSearch DSL Queries for searching and filtering large data sets without having to worry about formatting the JSON payloads. -Using intuitive search terms and operations, it's easier to construct human-readable search definitions without needing a deep +Using intuitive search terms and operations, it's easier to construct human-readable search definitions without needing a deep understanding of the Query DSL syntax. ### Complex search term query looks like with JSON notation -Probably the best way to demonstrate Kaping is to look at the traditional query construct, then the Kaping way. +Probably the best way to demonstrate Kaping is to look at the traditional query construct, then the Kaping way. #### Traditional JSON query + ```ruby def multi @@ -75,15 +75,16 @@ def multi } end ``` -In the above example the nested JSON can become quite a handful, Kaping solves this by making the code more human readable. +In the above example the nested JSON can become quite a handful, Kaping solves this by making the code more human-readable. + +#### What the query looks like with Kaping -#### What the query looks likes with Kaping ```ruby def multi - date = DateTime.now.strftime('%Y-%m-%d') - - q = DVLA::Kaping::Query.new('bool') + date = DateTime.now.strftime('%Y-%m-%d') + + q = DVLA::Kaping::Query.new('bool') q.must.match('fruit.destination', 'Market'). match_phrase('fruit.code', 'LA-MOP'). match_phrase('fruitStatus', 'Ripe'). @@ -113,19 +114,21 @@ Let's break that down further ```Ruby my_query = DVLA::Kaping::Query.new('bool') ``` -This is the starting point for creating a query definition by calling an instance of the Kaping:: Query + +This is the starting point for creating a query definition by calling an instance of the Kaping:: Query class and assigning it to a variable, e.g. 'my_query' -We then set the type of query we want. The common ones are 'bool' or 'match' +We then set the type of query we want. The common ones are 'bool' or 'match' depending on your search context. -If don't require a complex query you could do a very basic match query: +If you don't require a complex query you could do a very basic match query: ```Ruby my_query = DVLA::Kaping::Query.new('match_phrase', foo: 'Bar') my_query.to_json ``` -this is equivalent to writing this query in JSON + +This is equivalent to writing this query in JSON: ```Ruby my_query = { "query": @@ -137,43 +140,49 @@ my_query = { "query": With Kaping the JSON formation and formatting is taken care of with the common Ruby call .to_json -In the large example above, each line is a new search term definition. There are various different terms you can use depending on what +In the large example above, each line is a new search term definition. There are various different terms you can use depending on what functionality you require. You can group these terms in positive or negative boolean operations. #### Current sub list of terms are: - **match_phrase** - match documents that contain an exact phrase - **match** - full-text search on a specific document field -- **exist** - search for documents that contain a specific field. -- **wildcard** - match a wildcard pattern, such as He**o +- **exist** - search for documents that contain a specific field. +- **wildcard** - match a wildcard pattern, such as `He\*\*o` - **term** - search for exact term in a field. - **prefix** - search for terms that begin with a specific prefix -- **regex** - search for terms that match a regular expression, eg "[a-zA-Z]amlet" +- **regex** - search for terms that match a regular expression, eg `"[a-zA-Z]amlet"` - **between** - search for a range of values in a field -These are a mix of full-text queries and term-level queries, but they are most commonly use for our kind of searches, +These are a mix of full-text queries and term-level queries, but they are most commonly use for our kind of searches, other can easily be added as requirements dictate. The next part is the data field you want to search on -> DVLA::Kaping::Query.new('match_phrase', **foo**: 'Bar') +```ruby +DVLA::Kaping::Query.new('match_phrase', **foo**: 'Bar') +``` -Then the last part is the data you are looking for, the data type can be a range, exact match string, regex or a filter for example +Then the last part is the data you are looking for, the data type can be a range, exact match string, regex or a filter, for example: -> DVLA::Kaping::Query.new('match_phrase', foo: **' Bar'**) +```ruby +DVLA::Kaping::Query.new('match_phrase', foo: **' Bar'**) +``` ## Configuration + The Gem can be configured through a 'kaping.yml' file. Such configs as the logging level, result size and the AWS configs -can be set in the yaml, The config file can also be used to pick up any environment settings which is useful if running +can be set in the yaml, The config file can also be used to pick up any environment settings which is useful if running in a CI pipeline. - ## The Client -Currently we have a AWS OpenSearch client which takes care of the Sig4C signing. The AWS credentials can either be supplied + +Currently, we have an AWS OpenSearch client which takes care of the Sig4C signing. The AWS credentials can either be supplied as ENV variables or through a profile. ## Search -As long as your client is configure you can also use the optional built-in search facility + +As long as your client is configured you can also use the optional built-in search facility ```ruby my_query = DVLA::Kaping::Query.new('match_phrase', foo: 'Bar') @@ -186,15 +195,11 @@ response.dig('hits', 'hits') [External link to Kaping in rubygems](https://rubygems.org/gems/dvla-kaping) -We wanted to open source this Ruby Gem to a wider audiance so that others can also benefit from simplifying their OpenSearch queries. We also +We wanted to open source this Ruby Gem to a wider audience so that others can also benefit from simplifying their OpenSearch queries. We also embrace the feedback to further enhance the Gem and increase it's scope. - ## Further Development As mentioned before, not all the functionality of OpenSearch has been implemented, but any requests to expand will be taken into consideration. The code base for Kaping is small, the query builder is 40 lines of code. The separation of terms into their own module makes it easy to add additional query terms as required. - - - diff --git a/content/posts/2025-05-29-checking-for-atlas-users-before-creating/index.md b/content/posts/2025-05-29-checking-for-atlas-users-before-creating/index.md index f49b6ec4..39746ed7 100644 --- a/content/posts/2025-05-29-checking-for-atlas-users-before-creating/index.md +++ b/content/posts/2025-05-29-checking-for-atlas-users-before-creating/index.md @@ -10,65 +10,67 @@ ShowToc: true TocOpen: true --- -Ever had to create users in MongoDb Atlas in AWS for use when making database enquiries in a functional test written in Ruby? -Ever wished you could leave it to the test pack to check if one already exists and create one when required, instead of hand cranking a new one every single day of your life? +Ever had to create users in MongoDb Atlas in AWS for use when making database enquiries in a functional test written in Ruby? +Ever wished you could leave it to the test pack to check if one already exists and create one when required, instead of hand cranking a new one every single day of your life? -Well now you can!!! +Well now you can!!! -#### But, first things first +## But, first things first What is MongoDB Atlas and what do its users do? -Atlas is a repository of database users. A user assumes a role which has database access and permissions. +Atlas is a repository of database users. A user assumes a role which has database access and permissions. -Access could be to one or more databases. +Access could be to one or more databases. Permissions could be read, read/write, etc. -All the info you could ever want is here: https://www.mongodb.com/docs/atlas/ +All the info you could ever want is here: [MongoDB Atlas: Multi-Cloud Database Service](https://www.mongodb.com/docs/atlas/) -#### Why do this? Why not create a new user every time? +## Why do this? Why not create a new user every time? Because Atlas users sit outside of a test pack and can be used for the whole time they exist. Plus, creating & deleting users for every test is pointless and expensive, data-wise. -#### Is there anybody in there...? +## Is there anybody in there...? Assuming you have atlas installed and configured on your machine... Let's check the existing users: -In Terminal: +In Terminal: -```ruby +```shell atlas dbusers list ``` -This will return a list of the current users set up in the MongoDB Atlas instance you're connected to. - -The list will look something like this: +This will return a list of the current users set up in the MongoDB Atlas instance you're connected to. -USERNAME DATABASE -test-repo-admin-user admin -mongodb-realm-triggers_realmapp-service-name-here-cluster admin -mongodb-real-service-name-here-mongodb-atlas admin -arn:aws:iam:::role/ $external -arn:aws:iam:::role/ $external +The list will look something like this: +```shell +USERNAME DATABASE +test-repo-admin-user admin +mongodb-realm-triggers_realmapp-service-name-here-cluster admin +mongodb-real-service-name-here-mongodb-atlas admin +arn:aws:iam:::role/ $external +arn:aws:iam:::role/ $external +``` -Atlas also has a method which can be used to search for a specific user +Atlas also has a method which can be used to search for a specific user: -```atlas dbusers describe #{enquirier_username} --authDB \\ -o json``` +```shell +atlas dbusers describe #{enquirer_username} --authDB \\ -o json` +``` -which returns a boolean value. +Which returns a boolean value. -#### Code +## Code Add this to the env.rb in your test pack. -note: the 'LOG.info' commands are to output info to console. +> **Note:** the `LOG.info` commands are to output info to console. ```ruby - require 'date' date_today = Date.today.strftime('%Y-%m-%d') @@ -86,4 +88,5 @@ LOG.info { 'Creating enquirer user in Atlas: '.cyan + "'#{enquirier_username}'" system "atlas dbusers create readAnyDatabase -u #{enquirier_username} --awsIAMType ROLE --deleteAfter #{date_today}T23:59:59Z" else LOG.info { 'Enquiry user already created: '.cyan + "'#{enquirier_username}'" } -end``` \ No newline at end of file +end +``` diff --git a/content/posts/2025-06-09-double-splat-nil/index.md b/content/posts/2025-06-09-double-splat-nil/index.md index 7f0669d9..0bcdb68a 100644 --- a/content/posts/2025-06-09-double-splat-nil/index.md +++ b/content/posts/2025-06-09-double-splat-nil/index.md @@ -4,13 +4,13 @@ title: "TiL: Double Splat Nil in Ruby" description: "What is a Double Splat Nil and how does it help clamp down on unwanted named arguments" draft: false date: 2025-06-09 -tags: ["Ruby", "nil", 'double splat', "Today I Learned"] +tags: ["Ruby", "nil", "double splat", "Today I Learned"] categories: ["TIL", "Ruby"] ShowToc: false TocOpen: false --- -# How Ruby can do what you don't expect +## How Ruby can do what you don't expect Ruby is a very clever language. It will take the code you've written and do it's best to make sure it runs. Take for example the following method call: @@ -35,7 +35,7 @@ Interesting. At first glance it might appear that this method would print `Hello So what is going on here? Well, it's because when we call `method_one` we are incorrectly assuming that `param_two` is a keyword argument and are providing the name alongside the value. This is where Ruby tries to be clever as it transforms that into the hash `{:param_two=>" world"}`, which is not what we were expecting. There are a few ways you can deal with this, but the one this post is about is adding a double splat nil to the end of the list of arguments. -# Double Splat Nil +## Double Splat Nil `**nil` might look a bit strange if you haven't seen it before, so lets break it down. A double splat (`**`) is used to indicate that you are expecting a quantity of keyword arguments to be provided when a method is invoked. For example: @@ -48,6 +48,7 @@ double_splat(arg_one: 123, arg_two: 'Four Five Six', arg_three: 789) ``` That prints out the following hash: + ```ruby {:arg_one=>123, :arg_two=>"Four Five Six", :arg_three=>789} ``` @@ -67,4 +68,4 @@ method_two('Hello', param_two: ' world') => no keywords accepted (ArgumentError) ``` -Success! Now, we get an error raised saying something has been called with a keyword when it shouldn't have been which means we don't have to deal with unexpected hashes. \ No newline at end of file +Success! Now, we get an error raised saying something has been called with a keyword when it shouldn't have been which means we don't have to deal with unexpected hashes. diff --git a/content/posts/2026-02-01-generating-ai-audio/agentic_edit_diagram.png b/content/posts/2026-02-01-generating-ai-audio/agentic_edit_diagram.png deleted file mode 100644 index 992f97e2..00000000 Binary files a/content/posts/2026-02-01-generating-ai-audio/agentic_edit_diagram.png and /dev/null differ diff --git a/content/posts/2026-02-01-generating-ai-audio/audry.png b/content/posts/2026-02-01-generating-ai-audio/audry.png deleted file mode 100644 index e9e0331e..00000000 Binary files a/content/posts/2026-02-01-generating-ai-audio/audry.png and /dev/null differ diff --git a/content/posts/2026-02-01-generating-ai-audio/images/agentic_edit_diagram.jpg b/content/posts/2026-02-01-generating-ai-audio/images/agentic_edit_diagram.jpg new file mode 100644 index 00000000..0a9d979e Binary files /dev/null and b/content/posts/2026-02-01-generating-ai-audio/images/agentic_edit_diagram.jpg differ diff --git a/content/posts/2026-02-01-generating-ai-audio/images/audry.jpg b/content/posts/2026-02-01-generating-ai-audio/images/audry.jpg new file mode 100644 index 00000000..ab49e115 Binary files /dev/null and b/content/posts/2026-02-01-generating-ai-audio/images/audry.jpg differ diff --git a/content/posts/2026-02-01-generating-ai-audio/index.md b/content/posts/2026-02-01-generating-ai-audio/index.md index 26f85d29..1a220726 100644 --- a/content/posts/2026-02-01-generating-ai-audio/index.md +++ b/content/posts/2026-02-01-generating-ai-audio/index.md @@ -9,35 +9,38 @@ categories: ["AI"] ShowToc: false TocOpen: false --- -[You may prefer to listen to the audio version of this blog post.](audio_podcast.mp3) + +{{