[{"body":"Technical writing on software quality, test automation, performance engineering, and the tools and practices that help teams ship with confidence.\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/","section":"blog","tags":null,"title":"Blog"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/categories/","section":"categories","tags":null,"title":"Categories"},{"body":"Welcome to my site, I'm glad you're here! I'm using this site to share what I'm doing with mobile app development, and some general blogging.\nMy first app, Simply Today, is a daily planner for weekends, retirement days, and any day that needs a little shape — now available on the App Store for iOS, with Android coming soon.\nBefore this chapter of life, I spent years as a test automation architect and software quality engineer. I designed automated testing strategies across performance, API, and end-to-end testing, helping teams shift quality left and ship with more confidence. That background still shapes how I build: small, reliable, and thoughtfully tested.\nI still write about software quality and testing here on the blog, and I share what I'm learning as I build apps. I believe in practical, hands-on approaches to technology — you'll find many of my posts include working examples, screenshots, and links to repositories where you can try things yourself.\nWant to connect? Feel free to drop me a line if you have questions or would like to chat.\nThanks!\n--Dennis\n","link":"https://dennis-whalen-dot-com.vercel.app/","section":"","tags":null,"title":"Dennis-Whalen Dot Com"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/categories/gallery/","section":"categories","tags":null,"title":"Gallery"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/maestro/","section":"tags","tags":null,"title":"Maestro"},{"body":"I've recently been working on a personal project that has both mobile and web frontends. I wanted to include E2E tests, but I didn't want to spend a bunch of time getting all of that setup for web, iOS, and Android.\nI just wanted a handful of happy-path E2E tests for an app that could run on a desktop browser, mobile browser, and native mobile.\nMost importantly, I wanted to get this running quickly so I could focus on actually building the app. That's when I found an open source tool called Maestro.\nWhat immediately caught my attention with Maestro is that it's so easy to get setup, and it handles both web and mobile with the same tool and syntax.\nHere's What a Test Looks Like Maestro tests are written in YAML. Here's a simple desktop browser example that searches DuckDuckGo:\n1url: https://duckduckgo.com 2--- 3- launchApp 4- tapOn: 5 text: \u0026#39;Search without being tracked\u0026#39; 6- inputText: \u0026#39;Maestro e2e testing\u0026#39; 7- pressKey: Enter 8- assertVisible: \u0026#34;.*Maestro is an open-source framework.*\u0026#34; Pretty straightforward, right? It opens DuckDuckGo, taps the search box, searches for \u0026quot;Maestro e2e testing\u0026quot;, and verifies that the results contain \u0026quot;Maestro is an open-source framework\u0026quot;. Note that for partial text matching, Maestro uses regex—the .* pattern means \u0026quot;any characters\u0026quot;, so \u0026quot;.*text.*\u0026quot; effectively does a \u0026quot;contains\u0026quot; match.\nTo be honest, I was not super excited to work with a tool that uses YAML to define the tests. In my regular job I spend a lot of time building out code-based automation suites, and that usually feels like the \u0026quot;right\u0026quot; way to do it. But is that always the case?\nMy personal project is not super complex, and I don't have a team of test automation folks. I have one dev and one QA, and they are both me. I want E2E tests, but I want to focus the majority of my time on building the app, not building fancy-pants automation frameworks.\nLet's run this test!\nSetup I am not assuming that everyone uses a Mac, but that's what I'm using so keep that in mind if you're reading this as a Windows or Unix person. Maestro is cross-platform, but some of the install steps will be different. See their setup documentation for more details.\nFirst, let's install Maestro. Open your terminal and run:\n1curl -fsSL \u0026#34;https://get.maestro.mobile.dev\u0026#34; | bash Or you can use Homebrew:\n1brew tap mobile-dev-inc/tap 2brew install maestro Verify it worked:\n1maestro --version OK so what do I need to install next? Huh, that's it?? Well then... let's run the test!\nRunning a Test 1maestro test flows/duckduckgo-search-desktop.yaml Maestro will open a browser, run through the test steps, and show you the results. If something fails, the output helps you figure out what went wrong, and you'll also get some detailed log files. Hopefully your run will look like this:\nRunning the Same Test on Mobile Browser You can run a similar test on a mobile browser. Here's the mobile version:\n1appId: com.android.chrome 2--- 3- launchApp 4- tapOn: \u0026#34;Search or type URL\u0026#34; 5- inputText: \u0026#34;https://duckduckgo.com\u0026#34; 6- pressKey: Enter 7- tapOn: 8 id: \u0026#34;searchbox_input\u0026#34; 9- inputText: \u0026#34;Maestro e2e testing\u0026#34; 10- pressKey: Enter 11- assertVisible: \u0026#34;.*Maestro is an open-source framework.*\u0026#34; Notice how the syntax is almost identical. The main difference is using url: for desktop browsers and appId: for mobile browsers. Other than that, Maestro uses the same commands for both.\nTo run this, you'll need an Android emulator. If you have Android Studio installed, you can use the AVD Manager to create one. Make sure Chrome is installed on the emulator (it usually is by default).\nOnce your emulator is running, just run the test:\n1maestro test flows/duckduckgo-search-mobile.yaml Hopefully you'll see the same interactions that you saw with the desktop browser test, and the same green results, like this!\nYou now have a taste for browser-based Maestro testing on a desktop browser and a mobile browser. Let's move away from the browser and use Maestro test a mobile app.\nTesting a Native Mobile App The built-in Android Contacts app is perfect for this because it's available on every Android device and works great in an emulator. Notice how the syntax is the same as the web test. Maestro uses the same commands whether you're testing web or native mobile.\nHere's a test that creates a new contact:\n1appId: com.google.android.contacts 2jsEngine: graaljs 3--- 4- evalScript: ${output.firstName = faker.name().firstName()} 5- evalScript: ${output.lastName = faker.name().lastName()} 6- evalScript: ${output.phoneNumber = faker.phoneNumber().phoneNumber()} 7- launchApp 8- tapOn: \u0026#34;Create contact\u0026#34; 9- tapOn: \u0026#34;First name\u0026#34; 10- inputText: ${output.firstName} 11- tapOn: \u0026#34;Last name\u0026#34; 12- inputText: ${output.lastName} 13- longPressOn: \u0026#34;Phone (Mobile)\u0026#34; 14- tapOn: \u0026#39;Select All\u0026#39; 15- eraseText 16- inputText: ${output.phoneNumber} 17- tapOn: \u0026#34;Save\u0026#34; 18- assertVisible: ${output.firstName + \u0026#34; \u0026#34; + output.lastName} 19- scrollUntilVisible: 20 element: \u0026#34;Delete\u0026#34; 21- tapOn: \u0026#34;Delete\u0026#34; 22- tapOn: \u0026#34;Delete\u0026#34; 23- assertVisible: \u0026#34;1 contact deleted\u0026#34; This test is a bit more advanced as it demonstrates Maestro's ability to generate dynamic test data using Faker. The jsEngine: graaljs setting enables JavaScript execution, and the evalScript commands at the top use Faker to generate random first names, last names, and phone numbers. These values are stored in the output object and referenced throughout the test using ${output.variableName} syntax.\nThis is just one example of integrating JavaScript with Maestro scripts. More detail can be found here.\nRunning It With your emulator running, execute:\n1maestro test flows/contacts-app-android.yaml The test will run, and you'll see the emulator actually perform the actions. If it passes, you'll see a nice success message. If it fails, Maestro will tell you what went wrong and where. Here's what I see:\nMaestro MCP MCP (Model Context Protocol) is a standardized protocol that bridges tools (like Maestro) to LLMs (like Claude or ChatGPT). Think of it as a universal connector that lets these AI models access and interact with your development tools.\nWhy it matters: If you're using these LLMs in your development workflow, Maestro includes an MCP that lets them interact with Maestro directly. They can read your test files, understand your test structure, suggest improvements, or even generate tests based on your app's behavior.\nHow to use it: The MCP server comes bundled with Maestro. To use it in Cursor:\nOpen Cursor Settings Navigate to the MCP section Click \u0026quot;Add new MCP Server\u0026quot; Configure it with: 1{ 2 \u0026#34;mcpServers\u0026#34;: { 3 \u0026#34;maestro\u0026#34;: { 4 \u0026#34;command\u0026#34;: \u0026#34;maestro\u0026#34;, 5 \u0026#34;args\u0026#34;: [\u0026#34;mcp\u0026#34;] 6 } 7 } 8} Save and restart Cursor Similar functionality is available in other tools like VS Code through MCP extensions. Once connected, the AI assistant can discover your Maestro flows, understand your test structure, and help you write better tests.\nMore details can be found here.\nA few things I didn't cover but want to mention Maestro is easy to run on you CI platform, and also has a Cloud plan. More info here.\nMaestro has a ton of sample flows to help you learn more here.\nMaestro has an IDE to help with identifying UI elements, generating code, and running commands. Check it out.\nTake a look at docs.maestro.dev for more examples, advanced features like nested flows and conditions, page objects, and tips for structuring larger test suites.\nHappy building and testing. Peace out!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/maestro/","section":"blog","tags":["maestro"],"title":"Maestro: A single framework for mobile and web E2E testing"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/","section":"tags","tags":null,"title":"Tags"},{"body":"Intro In my previous promptfoo post, we covered the basics of testing LLM prompts with simple examples using promptfoo. But when you're building an actual application that processes user-generated content at scale, you might discover that your carefully crafted prompt needs to handle far more complexity than you initially anticipated.\nMany teams are still doing \u0026quot;vibe testing\u0026quot; - manually checking a few examples, tweaking prompts based on gut feel, and hoping everything works in production. While this might get you started, a systematic evaluation framework puts you significantly ahead of the curve when it comes to building and maintaining reliable AI systems, and provides a mechanism to build a set of repeatable automated regression tests.\nOur Assignment Let's consider an example. You're working with a major ecommerce client, and your team is building a feature that will analyze user submitted product reviews. Your application needs to evaluate the product reviews, classify sentiment, extract key product features mentioned, detect potentially fake reviews, and make moderation decisions. This will help customers find trustworthy reviews and help your business maintain review quality.\nThe core of this system is a prompt that takes each incoming review and returns structured data, such as sentiment classification, confidence scores, extracted features, fake review indicators, and moderation recommendations.\nThis prompt might work well during development, but once deployed, it needs to handle the messy reality of real user reviews. Your prompt will definitely need to be able to handle things like:\nMixed sentiment reviews (loved the product, hated the shipping) Fake or suspicious reviews Reviews with profanity or inappropriate content Sarcastic or nuanced language Reviews that mention competitors This is where a systematic process with multiple scenarios becomes crucial.\nOur Requirements Speaking of systematic processes, before we dive into building our prompt and setting up the prompfoo tests, let's outline what the requirements would look like. We'll use our old friend gherkin.\n1Feature: Product Review Analysis Prompt 2 3 Scenario Outline: Prompt analyzes product reviews correctly 4 Given a product review analysis prompt 5 And a \u0026#34;\u0026lt;review_type\u0026gt;\u0026#34; product review 6 When the prompt processes the review 7 Then the sentiment should be classified as \u0026#34;\u0026lt;expected_sentiment\u0026gt;\u0026#34; 8 And fake review indicators should be \u0026#34;\u0026lt;fake_indicators\u0026gt;\u0026#34; 9 And the recommendation should be \u0026#34;\u0026lt;expected_recommendation\u0026gt;\u0026#34; 10 And key features should be extracted 11 12 Examples: 13 | review_type | expected_sentiment | expected_fake_indicators | expected_recommendation | 14 | positive | positive | absent | approve | 15 | negative | negative | absent | approve | 16 | mixed | mixed | absent | flag_for_review | 17 | suspicious | positive | present | flag_for_review | Gherkin is just a way to describe requirements in plain language. In this case, we have four main test scenarios: positive reviews, negative reviews, mixed sentiment reviews, and suspicious/fake reviews.\nPromptfoo doesn't use gherkin, but I do, and it helps me think through the scenarios we need to cover. We'll translate these scenarios into actual promptfoo tests next.\nMoving Beyond Inline YAML: File-Based Organization In my last post we defined the entire test in YAML. Before diving into complex scenarios, let's improve our testing structure by moving prompts into separate files. This makes them easier to maintain, version control, and collaborate on.\nProject Structure 1promptfoo-product-reviews/ 2├── prompts/ 3│ └── analyze-review.txt 4├── test-data/ 5│ ├── positive-review.txt 6│ ├── negative-review.txt 7│ ├── mixed-review.txt 8│ └── suspicious-review.txt 9├── analyze-review-spec.yaml 10└── package.json Creating Our Review Analysis Prompt Let's first create a prompt specifically designed for ecommerce product review analysis:\nprompts/analyze-review.txt\n1You are an expert product review analyzer for an ecommerce platform. Analyze the following product review and provide a structured assessment. 2 3Product Review: 4{{review_text}} 5 6Provide your analysis in the following JSON format. Return ONLY the JSON object, no markdown code blocks, no explanations, no additional text: 7{ 8 \u0026#34;sentiment\u0026#34;: \u0026#34;positive|negative|mixed\u0026#34;, 9 \u0026#34;confidence\u0026#34;: 0.0-1.0, 10 \u0026#34;key_features_mentioned\u0026#34;: [\u0026#34;feature1\u0026#34;, \u0026#34;feature2\u0026#34;], 11 \u0026#34;main_complaints\u0026#34;: [\u0026#34;complaint1\u0026#34;, \u0026#34;complaint2\u0026#34;], 12 \u0026#34;main_praise\u0026#34;: [\u0026#34;praise1\u0026#34;, \u0026#34;praise2\u0026#34;], 13 \u0026#34;suspected_fake\u0026#34;: boolean, 14 \u0026#34;fake_indicators\u0026#34;: [\u0026#34;indicator1\u0026#34;, \u0026#34;indicator2\u0026#34;], 15 \u0026#34;recommendation\u0026#34;: \u0026#34;approve|flag_for_review|reject\u0026#34;, 16 \u0026#34;summary\u0026#34;: \u0026#34;Brief 1-2 sentence summary\u0026#34; 17} 18 19Focus on: 20- Accurate sentiment classification, especially for mixed reviews 21- Extracting specific product features mentioned 22- Identifying potential fake review indicators such as generic language without specific details, suspicious patterns, overly positive language, and extreme superlatives, overly negative language 23- Providing actionable moderation recommendations 24 25IMPORTANT: Return ONLY valid JSON. Do not wrap in markdown code blocks or add any other text. Test Scenarios: Real-World Product Reviews So that's the prompt we're going to test. Now let's create diverse test scenarios that represent what you'd actually encounter in production. You might make these up, or you might use some actual production reviews.\nScenario 1: Genuine Positive Review Example test-data/positive-review.txt\nI've been using these wireless earbuds for 3 months now and I'm really impressed. The battery life is excellent - I get about 6-7 hours of continuous listening, and the case gives me 2-3 full charges. The sound quality is crisp and clear, with good bass response for the price point. They stay comfortable in my ears during workouts and haven't fallen out once. The touch controls take some getting used to but work reliably once you learn them. Only minor complaint is that the case is a bit bulky for my small pockets, but that's a trade-off for the extra battery. Would definitely recommend for anyone looking for reliable wireless earbuds under $100.\nScenario 2: Detailed Negative Review test-data/negative-review.txt\nVery disappointed with these earbuds. The connection constantly drops out, especially when my phone is in my pocket or more than a few feet away. The battery life is nowhere near the advertised 8 hours - I'm lucky to get 4 hours before they die. The sound quality is muddy and lacks clarity, particularly in the mid-range frequencies. They're also uncomfortable for extended wear - my ears start hurting after about an hour. The touch controls are oversensitive and constantly trigger accidentally when I adjust them. For the price, I expected much better quality. I've had $20 earbuds that performed better than these. Returning them and looking for alternatives.\nScenario 3: Mixed Sentiment Review test-data/mixed-review.txt\nThese earbuds are a mixed bag. On the positive side, the sound quality is really good - clear highs, decent bass, and good overall balance. The build quality feels solid and they look premium. The battery life meets expectations at around 6 hours. However, there are some significant issues. The Bluetooth connection is unreliable - frequent dropouts and sometimes one earbud stops working randomly. The fit is also problematic for me - they tend to slip out during exercise despite trying all the included ear tips. Customer service was helpful when I contacted them about the connection issues, but the firmware update they suggested didn't solve the problem. Overall, great sound quality let down by connectivity and fit issues. Might work better for others but not ideal for my use case.\nScenario 4: Suspicious/Fake Review test-data/suspicious-review.txt\nAmazing product! These earbuds are the best I have ever used in my entire life. The sound quality is absolutely perfect and the battery life is incredible. They are so comfortable and never fall out. The connection is always stable and strong. I love everything about these earbuds and they exceeded all my expectations. Everyone should buy these right now because they are the greatest earbuds ever made. Five stars without any doubt! Highly recommend to all people who want amazing earbuds with perfect quality and performance.\nComprehensive Test Configuration Now let's create a promptfoo configuration that tests all these scenarios with appropriate assertions:\nanalyze-review-spec.yaml\n1description: Product Review Analysis Testing 2 3prompts: 4 - file://prompts/analyze-review.txt 5 6providers: 7 - openai:chat:gpt-4o-mini 8 9tests: 10 # Test 1: Genuine Positive Review 11 - vars: 12 review_text: file://test-data/positive-review.txt 13 assert: 14 - type: is-json 15 - type: javascript 16 value: | 17 const response = JSON.parse(output); 18 response.sentiment === \u0026#39;positive\u0026#39; \u0026amp;\u0026amp; response.confidence \u0026gt; 0.7 19 - type: contains-json 20 value: 21 suspected_fake: false 22 - type: llm-rubric 23 value: \u0026#34;Should identify key positive features like battery life, sound quality, and comfort. Should not flag as fake since it contains specific details and minor complaints.\u0026#34; 24 25 # Test 2: Detailed Negative Review 26 - vars: 27 review_text: file://test-data/negative-review.txt 28 assert: 29 - type: is-json 30 - type: javascript 31 value: | 32 const response = JSON.parse(output); 33 response.sentiment === \u0026#39;negative\u0026#39; \u0026amp;\u0026amp; response.confidence \u0026gt; 0.7 34 - type: contains-json 35 value: 36 suspected_fake: false 37 - type: llm-rubric 38 value: \u0026#34;Should identify specific complaints about connection, battery, sound quality, and comfort. Should extract main issues for product team review.\u0026#34; 39 40 # Test 3: Mixed Sentiment Review 41 - vars: 42 review_text: file://test-data/mixed-review.txt 43 assert: 44 - type: is-json 45 - type: javascript 46 value: | 47 const response = JSON.parse(output); 48 response.sentiment === \u0026#39;mixed\u0026#39; 49 - type: llm-rubric 50 value: \u0026#34;Should correctly identify mixed sentiment, extracting both positive aspects (sound quality, build) and negative aspects (connectivity, fit). This is the most challenging scenario for sentiment analysis.\u0026#34; 51 52 # Test 4: Suspicious/Fake Review 53 - vars: 54 review_text: file://test-data/suspicious-review.txt 55 assert: 56 - type: is-json 57 - type: contains-json 58 value: 59 suspected_fake: true 60 - type: javascript 61 value: | 62 const response = JSON.parse(output); 63 response.fake_indicators \u0026amp;\u0026amp; response.fake_indicators.length \u0026gt; 0 64 - type: llm-rubric 65 value: \u0026#34;Should detect fake review indicators: overly positive language, lack of specific details, generic praise, and extreme superlatives.\u0026#34; Understanding the Test Specification Let's break down what this test configuration accomplishes. We have four distinct tests that correspond to the four key scenarios mentioned above:\nTest 1: Genuine Positive Review - References positive-review.txt Test 2: Detailed Negative Review - References negative-review.txt Test 3: Mixed Sentiment Review - References mixed-review.txt Test 4: Suspicious/Fake Review - References suspicious-review.txt Each test loads its respective product review using the file:// syntax, which tells promptfoo to read the content from the specified file and inject it into the review_text variable in our prompt.\nMulti-Layered Assertions Notice that we're using multiple types of assertions for comprehensive validation:\nis-json - Ensures the output is valid JSON format contains-json - Checks for specific key-value pairs in the response javascript - Uses inline JavaScript for custom validation logic (like checking sentiment and confidence scores) llm-rubric - Uses an LLM to evaluate whether the output meets human-readable criteria The inline JavaScript assertions are particularly powerful for complex validation. For example:\n1const response = JSON.parse(output); 2response.sentiment === \u0026#39;positive\u0026#39; \u0026amp;\u0026amp; response.confidence \u0026gt; 0.7 This validates both the sentiment classification AND ensures the AI is confident in its assessment, helping us catch edge cases where the model might be uncertain.\nInstallation \u0026amp; Setup 1# Install as a dev dependency in your project 2npm install --save-dev promptfoo Run the test 1# Run the tests 2npx promptfoo eval -c promptfoo-product-reviews/analyze-review-spec.yaml --no-cache 3# View the results in web viewer 4npx promptfoo view -y Understanding the Results The web viewer has a lot going on, and I could do an entire walkthrough of its features. For now, let's focus on the key insights it provides into the test and evaluation results.\nThe results are displayed in a grid, and you can see our prompt in the first row. The 2nd row shows the results of our first scenario, the positive review.\nNote the prompt did a pretty good job at analyzing the review based on our requirements, and displays the actual response from the test:\n1{ 2 \u0026#34;sentiment\u0026#34;: \u0026#34;positive\u0026#34;, 3 \u0026#34;confidence\u0026#34;: 0.9, 4 \u0026#34;key_features_mentioned\u0026#34;: [\u0026#34;battery life\u0026#34;, \u0026#34;sound quality\u0026#34;, \u0026#34;comfort\u0026#34;, \u0026#34;touch controls\u0026#34;], 5 \u0026#34;main_complaints\u0026#34;: [\u0026#34;case is bulky\u0026#34;], 6 \u0026#34;main_praise\u0026#34;: [\u0026#34;excellent battery life\u0026#34;, \u0026#34;crisp and clear sound quality\u0026#34;, \u0026#34;comfortable during workouts\u0026#34;, \u0026#34;reliable touch controls\u0026#34;], 7 \u0026#34;suspected_fake\u0026#34;: false, 8 \u0026#34;fake_indicators\u0026#34;: [], 9 \u0026#34;recommendation\u0026#34;: \u0026#34;approve\u0026#34;, 10 \u0026#34;summary\u0026#34;: \u0026#34;The reviewer expresses high satisfaction with the wireless earbuds, highlighting their excellent battery life and sound quality while noting a minor complaint about the case size.\u0026#34; 11} Adding the tests to CI This is a great start, but we can take this a step further. Since promptfoo just runs from the command line, we can include it as a regression test in our CI pipeline and ensure that future prompt changes don't break these tests.\nIf we make changes to the prompt, or change the LLM provider, we can re-run this test and see if the results change. If they do, we can investigate why.\nAs requirements change and morph, we can adapt the tests accordingly.\nWrap-up In this post, we've explored how to set up a comprehensive testing framework for AI-generated product reviews using promptfoo. By defining clear test scenarios and leveraging multi-layered assertions, we can ensure our AI behaves as expected across a range of inputs.\nIt might not surprise you to learn that my prompt was not perfect the first time. Since I setup my automated tests first, it made it easy to iterate on the prompt development. Sounds like test driven development, huh?\nThat's it for now. Stay tuned for another promptfoo post before too long!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/ai/promptfoo/promptfoo-2-add-structured-testing-to-your-ai-prompt-vibe/","section":"blog","tags":["AI","promptfoo"],"title":"Add Structured Testing to Your AI Vibe - with promptfoo"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/ai/","section":"tags","tags":null,"title":"AI"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/series/promptfoo/","section":"series","tags":null,"title":"Promptfoo"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/promptfoo/","section":"tags","tags":null,"title":"Promptfoo"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/series/","section":"series","tags":null,"title":"Series"},{"body":"Intro On a recent client engagement, we needed a mechanism to validate LLM responses for an application that used AI to summarize customer service call transcripts.\nThe requirements were clear: each summary had to capture specific details (customer names, account numbers, actions taken, resolution details, etc.), and our validation process needed to be automated and repeatable. We needed to test our custom summarization prompts with the same rigor we apply to traditional software: pass/fail assertions, regression baselines, and systematic tracking.\nThat's where promptfoo came in. Promptfoo let us codify these requirements into automated tests and iterate on prompt improvements with confidence.\nWhy Testing LLM Responses Is Different (And Why You Should Care) As software engineers and quality professionals, we're used to deterministic systems where the same input always produces the same output. LLM responses break that assumption: the same prompt can yield different valid answers, so traditional assertion patterns are often insufficient.\nHere's the challenge: How can you verify a prompt's response is contextually accurate when the response can vary with every request?\nThe solution is to shift from testing exact outputs to testing output quality, accuracy, and safety. You need assertions that can evaluate whether a response contains required information, follows guidelines, and avoids harmful content, regardless of the exact wording.\nTraditional testing falls short with LLM prompt responses because:\nNon-deterministic responses: Same input, different valid outputs Context-dependent behavior: Quality depends on conversation history Safety concerns: Content filtering and moderation requirements Performance variability: Response times and costs fluctuate If you've been struggling with manual testing of AI features or relying on trial-and-error for prompt engineering, this guide will show you how promptfoo brings systematic testing to AI development.\nWhat is Promptfoo? Promptfoo is an open-source testing framework specifically designed to enable test-driven development for LLM applications with structured, automated evaluation of prompts, models, and outputs.\nKey capabilities:\nAssertion-based validation with pass/fail criteria familiar to QA engineers Side-by-side prompt comparison for A/B testing different prompts and approaches Automated regression testing to catch quality degradation CI/CD integration for your existing pipelines Multi-model support (OpenAI, Anthropic, Google, Azure, local models) Promptfoo brings familiar testing methodologies to AI development:\nTest-driven development instead of trial-and-error and/or hoping for the best Regression testing to catch quality degradation Performance monitoring (latency, cost, accuracy) Getting Started: Hands-On Examples The best way to understand promptfoo is to see it in action. Let's start with installation and work through practical examples.\nInstallation \u0026amp; Setup 1# Install as a dev dependency in your project 2npm install --save-dev promptfoo Configuration: YAML-Driven Testing Promptfoo uses YAML configuration files to define your tests. This approach will feel familiar if you've worked with other testing frameworks or CI/CD tools. The YAML file specifies:\nPrompts: The actual prompts you want to test Providers: Which AI models to use (OpenAI, Anthropic, Azure, etc.) Tests: Input variables and assertions used to validate responses Test scenarios: Different inputs and expected behaviors This declarative approach makes it easy to version control your AI tests and collaborate with your team.\nExample 1: Simple Dataset Generation Let's start with a simple example. We want to test a prompt that generates a list of random numbers. Of course an LLM is really not the right place to do this, but this is just for example purposes.\nWe're going to test this prompt against two different models: Claude and GPT-5-mini. (FYI, you will need API tokens for any paid model you are referencing.)\n1# examples-for-blog/ten_numbers.yaml 2description: Generating a random list of integers between a range 3 4prompts: 5 - \u0026#34;You are a JSON-only responder. OUTPUT EXACTLY one valid JSON array and NOTHING ELSE. Example: [10, 20, 30]. Generate an ordered list of ten random integers between {{start}} and {{end}} (inclusive). Use numeric values (no quotes), sorted in ascending order, and do not include any commentary or code fences.\u0026#34; 6 7providers: 8 - id: anthropic:messages:claude-3-haiku-20240307 9 - id: openai:chat:gpt-5-mini 10 11tests: 12 - vars: 13 start: 10 14 end: 1000 15 assert: 16 - type: is-json 17 value: | 18 { 19 \u0026#34;type\u0026#34;: \u0026#34;array\u0026#34;, 20 \u0026#34;minItems\u0026#34;: 10, 21 \u0026#34;maxItems\u0026#34;: 10, 22 \u0026#34;items\u0026#34;: { 23 \u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, 24 \u0026#34;minimum\u0026#34;: 10, 25 \u0026#34;maximum\u0026#34;: 1000 26 } 27 } How this works:\nWhen promptfoo runs this test, it substitutes the variables (start: 10 and end: 1000) into the prompt and sends it to both Claude and GPT-5-mini. Each model generates a response.\nThe is-json assertion is evaluated by promptfoo after it parses the model output as JSON. In other words, promptfoo performs the JSON parsing and schema validation (not the model). If the model returns something that isn't valid JSON or doesn't match the schema, the assertion will fail and promptfoo will report the parsing error and the schema mismatch.\nThis example demonstrates:\nVariable substitution with {{start}} and {{end}} Multiple model comparison (Claude vs GPT-5-mini) Programmatic validation using is-json so validation happens in promptfoo, not in the LLM Running the test is easy:\n1# run the test 2npx promptfoo eval -c examples-for-blog/ten_numbers.yaml To see a side-by-side comparison showing how each model performed and whether they passed the validation criteria:\n1# open the web report for the last run 2npx promptfoo view Here is our web view of the test results. Note you can see variables, prompts, model responses, validation outcomes, and even performance and cost metrics, all in one place. Example 2: Call Summary Validation (Real-World Use Case) So Example 1 was interesting, but let's look at how we can validate the output of a prompt by using an LLM to grade that output.\nHere's a more complex example based on our actual client engagement I described earlier - testing an AI system that summarizes customer service calls:\n1# examples-for-blog/customer-call-summary.yaml 2description: Call Summary Quality Testing 3 4prompts: 5 - | 6 Summarize this customer service call. Keep the summary succinct without unnecessary details. Pay special attention to include the agent\u0026#39;s demeanor and indicate if they ever seemed unprofessional. Include: 7 - Customer name and account number 8 - Issue description 9 - Actions taken by agent 10 - Any order number that is mentioned 11 - Resolution status 12 13 Call transcript: {{transcript}} 14 15providers: 16 - openai:chat:gpt-5-mini 17 18tests: 19 - vars: 20 transcript: | 21 Agent: Good morning, thank you for calling customer service. This is Maria, how can I help you today? 22 Customer: Hi Maria, I\u0026#39;m calling about an order I placed last week that was supposed to be delivered two days ago, but it still hasn\u0026#39;t arrived. 23 Agent: I\u0026#39;m sorry to hear about the delay with your order. I\u0026#39;d be happy to help you track that down. Can I start by getting your first and last name please? 24 Customer: Yes, it\u0026#39;s David Rodriguez. 25 Agent: Thank you Mr. Rodriguez. And can I also get your account number to verify your account? 26 Customer: Sure, it\u0026#39;s account number 78942. 27 Agent: Perfect, thank you. Now, can you provide me with the order number for the package you\u0026#39;re expecting? 28 Customer: Yes, the order number is ORD-2024-5583. 29 Agent: Great, and when did you place this order? 30 Customer: I placed it last Tuesday, January 16th. 31 Agent: Thank you for that information. Let me pull up your order details here... Okay, I can see order ORD-2024-5583 placed on January 16th, and you\u0026#39;re absolutely right - it was originally scheduled for delivery on January 22nd. I sincerely apologize for this delay, Mr. Rodriguez. 32 Customer: So what happened? Why didn\u0026#39;t it arrive when it was supposed to? 33 Agent: It looks like there was a sorting delay at our distribution center that affected several shipments in your area. Your package is currently in transit and I can see it\u0026#39;s now scheduled to be delivered this Friday, January 26th, by end of day. 34 Customer: Friday? That\u0026#39;s three days later than promised. This is really inconvenient. 35 Agent: I completely understand your frustration, and I apologize again for the inconvenience this has caused. To make up for the delay, I\u0026#39;m going to issue a $15 credit to your account, and I\u0026#39;ll also send you tracking information via email so you can monitor the package\u0026#39;s progress. 36 Customer: Okay, well I appreciate that. Will I get a notification when it\u0026#39;s actually delivered? 37 Agent: Absolutely. You\u0026#39;ll receive both an email and text notification once the package is delivered, and the tracking information will show real-time updates. Is there anything else I can help you with today? 38 Customer: No, that covers it. Thank you for your help, Maria. 39 Agent: You\u0026#39;re very welcome to never ever call me again, Mr. Rodriguez. Again, I apologize for the delay, and thank you for your patience. Have a great day! 40 assert: 41 - type: contains 42 value: \u0026#34;David Rodriguez\u0026#34; 43 - type: contains 44 value: \u0026#34;ORD-2024-5583\u0026#34; 45 - type: llm-rubric 46 value: \u0026#34;Summary should indicate whether the agent seemed professional or not, and should include all key details, including the action taken by the agent, the resolution, and any compensation offered.\u0026#34; This prompt embeds a long customer-service phone transcript that the model is asked to summarize succinctly while preserving key facts. To verify correctness we include a couple of deterministic assertions (exact-match checks) for the customer's name and the order number so those values must appear in the summary.\nWe also include an llm-rubric asset: promptfoo will call an LLM to grade the generated summary against the supplied rubric text, allowing us to assert on higher-level quality attributes such as professionalism, completeness, and whether the agent's actions and compensation were described.\nNow I can run that test and see how we do!!\n1# run the test 2npx promptfoo eval -c examples-for-blog/customer-call-summary.yaml 3# View results 4npx promptfoo view And here are our results: Note the prompt specifically requests to indicate the agent's demeanor, and we use the rubric to verify the output contains it. Since I never trust a test unless I can see it fail, I'm going to temporarily remove the mention of demeanor in the prompt, but leave the assert alone, so we should get a failure. Drumroll, please…\nAnd we do!\nAs you can see, the test caught the error with our prompt:\nConclusion I got a little long-winded with this post, but I hope someone out there finds it useful. Promptfoo represents a paradigm shift from manual AI testing to systematic, automated evaluation. By bringing familiar testing methodologies to AI development, it enables teams to build reliable, secure, and high-quality AI applications.\nI'll be back soon with some more promptfoo content, and you should certainly check out the awesome documentation at promptfoo.dev for excellent resources for getting started.\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/ai/promptfoo/promptfoo-1-testing-custom-llm-prompts/","section":"blog","tags":["AI","promptfoo"],"title":"Automate the Testing of Your LLM Prompt - with promptfoo"},{"body":"Website performance directly affects the user's experience, and your business's bottom line.\nOne way of identifying performance issues is via API-based load testing tools such as k6. API load tests tell you whether your services scale and how quickly they respond under load, but they don’t measure the full user experience.\nIf you focus only on load testing your backend, you might still ship a slow or jittery site because of render‑blocking CSS/JavaScript, heavy images/fonts, main‑thread work, layout shifts, and other front-end issues.\nUltimately users don't care where the performance issue resides, they just know your site is \u0026quot;slow\u0026quot;.\nThis slow performance can cost you customers, revenue, search visibility, and trust.\nWhat is Lighthouse? Lighthouse is an automated auditor built by Google and is part of the Chrome DevTools experience. While this post focuses on performance, Lighthouse also audits and provides actionable recommendations for accessibility, best practices, and SEO.\nHow Lighthouse works Launches Chrome and navigates to your page using the Chrome DevTools Protocol. Emulates device, network, and CPU to keep runs comparable. Records a performance trace and analyzes it against a set of audits. Outputs scores and detailed metrics with fix ideas. Can be included in your CI pipeline. Core Web Vitals: what they mean and why they matter These user‑focused metrics map to how fast content shows up, how responsive the page feels, and how stable it looks.\nCore Web Vitals at a glance Metric Plain meaning Good target What you’ll see in Lighthouse LCP (Largest Contentful Paint) Time to show the largest thing in the initial viewport (often the primary image or a big text block). ≤ 2.5 s LCP value in the Metrics section FID (First Input Delay) Delay from a user’s first tap/click to when the page can start handling it. In Lighthouse runs, use Total Blocking Time (TBT) as the responsiveness indicator. FID ≤ 100 ms; aim for low TBT TBT value in the Metrics section CLS (Cumulative Layout Shift) How much content unexpectedly moves while the page loads (visual stability). ≤ 0.1 CLS score in Metrics/Diagnostics Sample Lighthouse Report Regardless of how you run Lighthouse, you get a detailed report with scores, metrics, and prioritized suggestions.\nOverall scores: What went wrong? What looks good? Running Lighthouse Lighthouse can be run in a number of ways, including:\nChrome DevTools (UI) Command line (CLI) Node module (programmatic) Run Lighthouse from Chrome DevTools Open your site in Chrome → Right‑click Inspect → Lighthouse tab → Set your analysis options → Analyze. This generates a full HTML report inside DevTools.\nRun Lighthouse from the command line Install Lighthouse (requires Node.js):\n1npm install -g lighthouse Basic mobile audit and open the HTML report:\n1lighthouse https://www.demoblaze.com \\ 2 --output=html \\ 3 --output-path=./reports/lighthouse.html \\ 4 --view Export JSON for automation or tracking:\n1lighthouse https://www.demoblaze.com \\ 2 --output=json \\ 3 --output-path=./reports/lighthouse.json \\ 4 --chrome-flags=\u0026#34;--headless\u0026#34; Desktop profile:\n1lighthouse https://www.demoblaze.com --preset=desktop --output=html --output-path=./reports/desktop.html Use throttling to simulate slower networks:\n1lighthouse https://www.demoblaze.com \\ 2 --throttling-method=simulate \\ 3 --throttling.rttMs=150 \\ 4 --throttling.throughputKbps=1638.4 \\ 5 --throttling.cpuSlowdownMultiplier=4 \\ 6 --output=html --output-path=./reports/consistent.html Focus audits on key performance metrics with a config (lighthouse-config.js):\n1module.exports = { 2 extends: \u0026#39;lighthouse:default\u0026#39;, 3 settings: { 4 onlyAudits: [ 5 \u0026#39;first-contentful-paint\u0026#39;, 6 \u0026#39;largest-contentful-paint\u0026#39;, 7 \u0026#39;cumulative-layout-shift\u0026#39;, 8 \u0026#39;total-blocking-time\u0026#39; 9 ], 10 throttlingMethod: \u0026#39;simulate\u0026#39;, 11 throttling: { rttMs: 150, throughputKbps: 1638.4, cpuSlowdownMultiplier: 4 } 12 } 13}; Run with the config:\n1lighthouse https://www.demoblaze.com --config-path=./lighthouse-config.js --output=html --output-path=./reports/focused.html Programmatic usage (Node) Why use this? Programmatic runs let you script real user interactions and measure performance along a flow (navigations, clicks, route changes). With Puppeteer + Lighthouse User Flows you can drive the browser, capture metrics per step, and generate a single report—perfect for CI, regression checks, and measuring critical journeys like signup or checkout.\nNote: Lighthouse currently only supports Puppeteer for programmatic user flows.\nInstall packages:\n1npm i lighthouse puppeteer Save as user-flow.mjs and run with node user-flow.mjs:\n1import {writeFileSync, mkdirSync} from \u0026#39;fs\u0026#39;; 2import puppeteer from \u0026#39;puppeteer\u0026#39;; 3import {startFlow} from \u0026#39;lighthouse\u0026#39;; 4 5const browser = await puppeteer.launch({headless: \u0026#39;new\u0026#39;}); 6const page = await browser.newPage(); 7const flow = await startFlow(page); 8 9// Navigate to Demoblaze 10await flow.navigate(\u0026#39;https://www.demoblaze.com\u0026#39;); 11 12// Interaction-initiated navigation via a callback function 13await flow.navigate(async () =\u0026gt; { 14 await page.click(\u0026#39;a[href=\u0026#34;index.html\u0026#34;]\u0026#39;); 15}); 16 17// Start/End a navigation around a user action 18await flow.startNavigation(); 19await page.click(\u0026#39;a#cartur\u0026#39;); // open Cart 20await flow.endNavigation(); 21 22await browser.close(); 23mkdirSync(\u0026#39;./reports\u0026#39;, {recursive: true}); 24writeFileSync(\u0026#39;./reports/lh-flow-report.html\u0026#39;, await flow.generateReport()); 25console.log(\u0026#39;Saved ./reports/lh-flow-report.html\u0026#39;); Wrap‑up Start by running Lighthouse in DevTools (fast feedback) or the CLI (repeatable results). Focus on three things: LCP (how fast the main content shows), TBT (how responsive it feels), and CLS (how stable it looks).\nWhat’s next in this series:\nContainerize Lighthouse runs with Docker for consistent local and CI environments Add Lighthouse checks to a GitHub Actions workflow with performance budgets and PR comments Export key metrics to Prometheus for time‑series storage Visualize trends and budgets in a Grafana dashboard ","link":"https://dennis-whalen-dot-com.vercel.app/blog/performance-testing/automated-browser-based-performance-testing/","section":"blog","tags":["performance testing"],"title":"Automating Browser-Based Performance Testing"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/performance-testing/","section":"tags","tags":null,"title":"Performance Testing"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/load-testing/","section":"tags","tags":null,"title":"Load Testing"},{"body":"Introduction Load testing is crucial for ensuring your applications can handle expected load volumes. In this guide, we'll set up a complete load testing environment using k6 for testing, Prometheus for metrics collection, and Grafana for visualization—all orchestrated with Docker.\nAlthough there are paid versions of these products, this guide will focus exclusively on a basic setup with their open-source Docker images.\nPrerequisites Docker and Docker Compose installed Basic understanding of load testing concepts Familiarity with Docker Architecture Overview Our setup consists of four main components:\nk6: Executes load tests and exports metrics Application: A simple API-based application to test Prometheus: Collects and stores metrics from k6 Grafana: Visualizes metrics from Prometheus These components will be implemented with 4 Docker containers. Here's how these components interact:\nData Flow: Load generation: our k6 script sends HTTP requests to the Sample API to simulate user traffic Metrics Export: as the test runs, performance metrics from k6 are exported to Prometheus via remote write Data Query: Grafana uses PromQL to query Prometheus for metrics All components run within the same Docker network, enabling seamless communication between services.\nProject Structure 1k6-prometheus-grafana/ 2├── docker-compose.yml 3├── prometheus/ 4│ └── prometheus.yml 5├── grafana/ 6│ └── dashboards/ 7│ └── k6-dashboard.json 8├── k6/ 9│ └── script.js 10└── sample-api/ 11 └── Dockerfile 12 └── server.js Step 1: Create the Sample API First, let's create a simple Node.js API to test against:\nsample-api/server.js\n1const express = require(\u0026#39;express\u0026#39;); 2const app = express(); 3 4app.get(\u0026#39;/health\u0026#39;, (req, res) =\u0026gt; { 5 res.json({ status: \u0026#39;healthy\u0026#39;, timestamp: new Date().toISOString() }); 6}); 7 8app.get(\u0026#39;/api/users/:id\u0026#39;, (req, res) =\u0026gt; { 9 const { id } = req.params; 10 // Simulate some processing delay 11 setTimeout(() =\u0026gt; { 12 res.json({ id, name: `User ${id}`, timestamp: new Date().toISOString() }); 13 }, Math.random() * 100); 14}); 15 16app.post(\u0026#39;/api/users\u0026#39;, (req, res) =\u0026gt; { 17 // Simulate user creation 18 setTimeout(() =\u0026gt; { 19 res.status(201).json({ 20 id: Math.floor(Math.random() * 1000), 21 message: \u0026#39;User created successfully\u0026#39; 22 }); 23 }, Math.random() * 200); 24}); 25 26app.listen(3000, () =\u0026gt; { 27 console.log(\u0026#39;Server running on port 3000\u0026#39;); 28}); And add a docker file that will start the app:\nsample-api/Dockerfile\n1FROM node:16-alpine 2WORKDIR /app 3RUN npm init -y \u0026amp;\u0026amp; npm install express 4COPY . . 5EXPOSE 3000 6CMD [\u0026#34;node\u0026#34;, \u0026#34;server.js\u0026#34;] Step 2: Create k6 Test Script This JavaScript test script defines how k6 will interact with our sample API during the load test.\nk6/script.js\n1import http from \u0026#39;k6/http\u0026#39;; 2import { check, sleep } from \u0026#39;k6\u0026#39;; 3import { Rate, Counter, Trend } from \u0026#39;k6/metrics\u0026#39;; 4 5// Custom metrics - these allow us to track specific aspects of our test 6export const errorRate = new Rate(\u0026#39;errors\u0026#39;); // Tracks percentage of errors 7export const myCounter = new Counter(\u0026#39;my_counter\u0026#39;); // Simple incrementing counter 8export const responseTime = new Trend(\u0026#39;response_time\u0026#39;); // Tracks response time distribution 9 10export const options = { 11 stages: [ 12 { duration: \u0026#39;30s\u0026#39;, target: 5 }, // Ramp up to 5 virtual users over 30 seconds 13 { duration: \u0026#39;90s\u0026#39;, target: 20 }, // Ramp to from 5 to 20 virtual users over 90 seconds 14 { duration: \u0026#39;3m\u0026#39;, target: 20 }, // Stay at 20 virtual users for 3 minutes 15 { duration: \u0026#39;30s\u0026#39;, target: 0 }, // Gradually ramp down to 0 over 30 seconds 16 ], 17 thresholds: { 18 http_req_duration: [\u0026#39;p(95)\u0026lt;500\u0026#39;], // 95% of requests must complete in less than 500ms for the test to pass 19 http_req_failed: [\u0026#39;rate\u0026lt;0.1\u0026#39;], // Test fails if more than 10% of requests fail 20 }, 21}; 22 23export default function () { 24 const baseUrl = \u0026#39;http://sample-api:3000\u0026#39;; 25 26 // Test GET endpoint - fetches a random user 27 let getResponse = http.get(`${baseUrl}/api/users/${Math.floor(Math.random() * 100)}`); 28 check(getResponse, { 29 \u0026#39;GET status is 200\u0026#39;: (r) =\u0026gt; r.status === 200, 30 \u0026#39;GET response time \u0026lt; 500ms\u0026#39;: (r) =\u0026gt; r.timings.duration \u0026lt; 500, 31 }); 32 33 // Track custom metrics for this request 34 errorRate.add(getResponse.status !== 200); 35 responseTime.add(getResponse.timings.duration); 36 myCounter.add(1); 37 38 sleep(1); // Pause for 1 second between requests 39 40 // Test POST endpoint - creates a new user 41 let postResponse = http.post(`${baseUrl}/api/users`, JSON.stringify({ 42 name: `TestUser_${Date.now()}`, 43 email: `test_${Date.now()}@example.com` 44 }), { 45 headers: { \u0026#39;Content-Type\u0026#39;: \u0026#39;application/json\u0026#39; }, 46 }); 47 48 check(postResponse, { 49 \u0026#39;POST status is 201\u0026#39;: (r) =\u0026gt; r.status === 201, 50 \u0026#39;POST response time \u0026lt; 1000ms\u0026#39;: (r) =\u0026gt; r.timings.duration \u0026lt; 1000, 51 }); 52 53 errorRate.add(postResponse.status !== 201); 54 myCounter.add(1); 55 56 sleep(1); 57} Step 3: Configure Prometheus Prometheus is an open-source monitoring and alerting toolkit that collects and stores time-series metrics. The configuration below sets up Prometheus to scrape metrics from both itself and the k6 load testing tool.\nprometheus/prometheus.yml\n1global: 2 scrape_interval: 15s # How frequently to scrape targets by default 3 evaluation_interval: 15s # How frequently to evaluate rules 4scrape_configs: 5 - job_name: \u0026#39;prometheus\u0026#39; # Self-monitoring configuration 6 static_configs: 7 - targets: [\u0026#39;localhost:9090\u0026#39;] # Prometheus\u0026#39;s own metrics endpoint 8 - job_name: \u0026#39;k6\u0026#39; # Configuration to scrape k6 metrics 9 static_configs: 10 - targets: [\u0026#39;k6:6565\u0026#39;] # k6\u0026#39;s metrics endpoint (using Docker service name) 11 scrape_interval: 5s # More frequent scraping for k6 during tests 12 metrics_path: /metrics # Path where metrics are exposed Once Prometheus is collecting metrics, we'll be able to query this data directly or visualize it through Grafana in the next steps.\nStep 4: Grafana Dashboard Configuration Create a dashboard provisioning file for automatic setup:\ngrafana/dashboards/dashboard.yml\n1apiVersion: 1 2 3providers: 4 - name: \u0026#39;default\u0026#39; 5 orgId: 1 6 folder: \u0026#39;\u0026#39; 7 type: file 8 disableDeletion: false 9 editable: true 10 options: 11 path: /etc/grafana/provisioning/dashboards Step 5: Docker Compose Configuration docker-compose.yml\n1services: 2 # Sample API service to be load tested by k6 3 sample-api: 4 build: ./sample-api 5 ports: 6 - \u0026#34;3000:3000\u0026#34; # Exposes API on localhost:3000 7 networks: 8 - k6-net 9 10 # Prometheus for metrics collection 11 prometheus: 12 image: prom/prometheus:latest 13 container_name: prometheus 14 ports: 15 - \u0026#34;9090:9090\u0026#34; # Prometheus UI available at localhost:9090 16 volumes: 17 - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml # Custom config 18 command: 19 - \u0026#39;--config.file=/etc/prometheus/prometheus.yml\u0026#39; 20 - \u0026#39;--storage.tsdb.path=/prometheus\u0026#39; 21 - \u0026#39;--web.console.libraries=/etc/prometheus/console_libraries\u0026#39; 22 - \u0026#39;--web.console.templates=/etc/prometheus/consoles\u0026#39; 23 - \u0026#39;--web.enable-lifecycle\u0026#39; # Allows config reloads without restart 24 - \u0026#39;--web.enable-remote-write-receiver\u0026#39; # Enables remote write endpoint for k6 25 networks: 26 - k6-net 27 28 # Grafana for dashboarding and visualization 29 grafana: 30 image: grafana/grafana:latest 31 container_name: grafana 32 ports: 33 - \u0026#34;3001:3000\u0026#34; # Grafana UI available at localhost:3001 34 environment: 35 - GF_SECURITY_ADMIN_PASSWORD=admin # Default admin password 36 volumes: 37 - grafana-storage:/var/lib/grafana # Persistent storage for Grafana data 38 - ./grafana/dashboards:/etc/grafana/provisioning/dashboards # Pre-provisioned dashboards 39 networks: 40 - k6-net 41 depends_on: 42 - prometheus # Waits for Prometheus to be ready 43 44 # k6 load testing tool with Prometheus remote write output 45 k6: 46 image: grafana/k6:latest 47 container_name: k6 48 ports: 49 - \u0026#34;6565:6565\u0026#34; 50 environment: 51 - K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write # Prometheus remote write endpoint 52 - K6_PROMETHEUS_RW_TREND_STATS=p(95),p(99),min,max # Custom trend stats 53 volumes: 54 - ./k6:/scripts # Mounts local k6 scripts 55 command: run --out experimental-prometheus-rw /scripts/script.js # Runs the main k6 script 56 networks: 57 - k6-net 58 depends_on: 59 - sample-api 60 - prometheus 61 62volumes: 63 grafana-storage: # Named volume for Grafana data 64 65networks: 66 k6-net: 67 driver: bridge # Isolated network for all services Step 6: Start the stack Start all services: 1docker-compose up -d Step 7: Setting Up a Pre-built K6 Dashboard Access Grafana: Navigate to http://localhost:3001 Login: Use admin/admin (you'll be prompted to change the password) Add Prometheus Data Source First: Go to Configuration → Data Sources Click \u0026quot;Add data source\u0026quot; Select \u0026quot;Prometheus\u0026quot; Set URL to: http://prometheus:9090 Click \u0026quot;Save \u0026amp; Test\u0026quot; Import K6 Dashboard: Click the \u0026quot;+\u0026quot; icon in the left sidebar Select \u0026quot;Import\u0026quot; Use one of these dashboard IDs for Prometheus: 19665 - K6 Prometheus (recommended) 10660 - K6 Load Testing Results (Prometheus) 19634 - K6 Performance Test Dashboard Click \u0026quot;Load\u0026quot; Select your Prometheus data source Click \u0026quot;Import\u0026quot; Step 8: Run the load test Run the k6 test: 1docker-compose run --rm k6 run --out experimental-prometheus-rw /scripts/script.js As the test runs, k6 will send API requests to the sample API, and metrics will be collected and sent to Prometheus. You can monitor the test progress in the terminal.\nStep 9: Monitor your test run in Grafana The Grafana UI available at http://localhost:3001. Select your Dashboard from the left nav and you can monitor your test real time with the Grafana dashboard, which should look something like this: Cleanup Stop and remove all containers and volumes:\n1docker-compose down -v Conclusion The point of this post was just to provide awareness of the open-source options available to you as you consider k6 for load testing. I skimmed over a lot of detail and explanation about k6, Prometheus, and Grafana. I will likely fill in some detail with future posts. Until then, this setup provides a complete observability stack for K6 load testing.\nThe Docker-based approach ensures consistency across environments and makes it easy to integrate into CI/CD pipelines. And FYI, you can find all the code from this blog post here.\nThanks for reading and let me know if you have any questions or suggestions for future posts!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/performance-testing/load-testing-with-k6-prometheus-grafana/","section":"blog","tags":["load testing","performance testing"],"title":"Load Testing with k6, Prometheus, and Grafana"},{"body":"Introduction In my previous post, I showed you how to use the axe DevTools chrome extension to test for accessibility issues on a webpage. Today, I'm going to show you how to do the same thing with Playwright, enabling you to automate accessibility testing in your CI/CD pipeline. Everything I'm going to demo can also be found in my sample repo.\nThe axe-core package The axe-core package is a JavaScript library that can be used to run accessibility tests on a webpage. It's the same library that powers the axe DevTools extension that we looked at in my previous post, but it can be run programmatically in a variety of environments.\nWe're going to use it with Playwright, but axe-core packages exist for a number of automated testing frameworks, including Cypress, Selenium, WebdriverIO, and more.\nOur first Playwright accessibility test Including accessibility tests in your Playwright suite is as simple as adding a few lines of code. We've got a sample website that we're going to test, and we're going to use Playwright to navigate to the page and run an accessibility check.\nHere's an example test that navigates to the webpage and runs an accessibility check:\n1import { test, expect } from \u0026#39;@playwright/test\u0026#39;; 2import AxeBuilder from \u0026#39;@axe-core/playwright\u0026#39;; 3 4test.describe(\u0026#39;Accessibility University testing\u0026#39;, () =\u0026gt; { 5 test(\u0026#39;Full page scan should not find accessibility issues\u0026#39;, async ({ page }) =\u0026gt; { 6 await page.goto(\u0026#39;https://www.washington.edu/accesscomputing/AU/before.html\u0026#39;); 7 await page.waitForLoadState(\u0026#39;networkidle\u0026#39;); 8 const accessibilityScanResults = await new AxeBuilder({ page }).analyze(); 9 expect(accessibilityScanResults.violations).toEqual([]); 10 }); 11}); A few things to note:\nwaitForLoadState('networkidle') waits for the page to finish loading before running the accessibility check. This is important because we want to make sure the page is fully rendered before we check for accessibility issues. new AxeBuilder({ page }).analyze() runs the accessibility check on the page and returns the results. expect(accessibilityScanResults.violations).toEqual([]); indicates that we are expecting 0 accessibility violations. If violations are found, this test will fail. Since our sample website is specifically designed to have lots of accessibility issues, we're expecting this test to fail, and it does!\nReporting OK so we know that our test is failing, but what accessibility issues are being found? The accessibilityScanResults object contains a lot of information about the accessibility issues that were found, and it's pretty easy to create a report with tweaks like this:\n1import { test, expect } from \u0026#39;@playwright/test\u0026#39;; 2import path from \u0026#39;path\u0026#39;; 3import AxeBuilder from \u0026#39;@axe-core/playwright\u0026#39;; 4import { createHtmlReport } from \u0026#39;axe-html-reporter\u0026#39;; 5 6test.describe(\u0026#39;Accessibility University testing\u0026#39;, () =\u0026gt; { 7 test(\u0026#39;Full page scan should of BEFORE page\u0026#39;, async ({ page }) =\u0026gt; { 8 await page.goto(\u0026#39;https://www.washington.edu/accesscomputing/AU/before.html\u0026#39;); 9 await page.waitForLoadState(\u0026#39;networkidle\u0026#39;); 10 const accessibilityScanResults = await new AxeBuilder({ page }).analyze(); 11 createHtmlReport({ 12 results: accessibilityScanResults, 13 options: { 14 outputDir: path.join(\u0026#39;e2e\u0026#39;, \u0026#39;test-results\u0026#39;, \u0026#39;accessibility-results\u0026#39;), 15 reportFileName: `my-report.html`, 16 }, 17 }); 18 expect(accessibilityScanResults.violations).toEqual([]); 19 }); 20}); createHtmlReport is imported from the axe-html-reporter package, and it creates an HTML report of the accessibility issues found. The report is saved to the e2e/test-results/accessibility-results directory with the name my-report.html. Here's a snippet of what the report for our test looks like:\nThis is just the first page of the report, and there are lots of details in following pages. You'll notice that there are 50 total violations, which matches what we saw when we used on the axe DevTools extension in my previous post.\nThe report also provides a detailed breakdown of the violations, including the rule that was violated, the impact of the violation, and a description of the issue.\nJust as we did with the Chrome extension, we can use this report to identify and fix the accessibility issues on our website.\nAdding your tests to the CI/CD pipeline If you have some general familiarity with Playwright, you probably already know how to run your tests in your CI/CD pipeline. If you don't, you can check out the Playwright documentation for more information. Since these accessibility tests are just Playwright tests, you can run them in the same way you run your other Playwright tests.\nConclusion In this post I showed you a basic example of how to include accessibility tests in your Playwright suite. Don't forget that axe-core is not limited to Playwright, and can be used with a variety of automated testing frameworks, such as Cypress and Selenium.\nSome additional things to note:\nAlthough I was just testing in Chrome, axe-core supports all major browsers. You should consider testing in multiple browsers and viewports to ensure that your website is accessible to all users with all devices.\nThe scan can be configured to include or exclude certain rules, for example color-contrast rules.\nYou can limit the scan to a subset of the page by using the include and exclude options.\nYou can limit the scan to a subset of the WCAG guidelines by using the rules option.\nFinally, it's probably appropriate to mention that a clean accessibility scan is a great first step in making your website accessible, but it's not a silver bullet. For example, take the test to verify the existence of alt-text for images. The tool can tell you if an image is missing alt-text, but it can't tell you if the alt-text is meaningful.\nManual validation via a screen reader is an additional step you can do to ensure that your website is accessible. Automation tooling exists that can allow you to automate the manual screen reader testing process, but that's a topic for another post. Stay tuned!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/accessibility-testing/accessibility-testing-with-playwright/","section":"blog","tags":["accessibility-testing"],"title":"Accessibility Testing with Playwright"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/accessibility-testing/","section":"tags","tags":null,"title":"Accessibility-Testing"},{"body":"Introduction An accessible website ensures that all users, regardless of their physical or cognitive limitations, can navigate and interact with its content.\nAccessibility addresses issues faced by people with impairments, such as:\nvisual impairments (requiring screen readers or high contrast) hearing impairments (requiring captions for audio content) mobility challenges (navigating without a mouse) cognitive disabilities (requiring clear and consistent layouts). Web Content Accessibility Guidelines are the gold standard for web accessibility, offering a comprehensive set of standards to ensure websites are usable by everyone.\nEven with these standards, testing for accessibility can be challenging, particularly when done manually. The good news? Many aspects of accessibility testing can be automated.\nLet's get started with a Chrome browser extension, and a website that could really use some help with accessibility.\naxe DevTools extension for Chrome A number of browser extensions are available to help you test for accessibility issues. One of the most popular is the \u0026quot;axe DevTools\u0026quot; extension for Chrome. This tool allows you to run accessibility checks right in the browser and view the results in an easy-to-understand format.\naxe DevTools unlocks more features with a Pro subscription, but the free version is still quite powerful. Everything we'll be doing here can be done with the free version.\nEnough chatter, let's see how this works!\nThe website I want to take a look at this page and see what kind of accessibility issues I can find. The page is specifically designed to have lots of accessibility issues, so it's a great place to start. It looks like this:\nInstall the extension Install the axe DevTools extension from the Chrome Web Store. Once installed, you can start it by selecting axe DevTools from the Chrome Developer Tools menu: And it will load, like this!\nRunning a scan Once you have the extension loaded, you can click the \u0026quot;Full Page Scan\u0026quot; button to run a scan on the entire page. This will check for a variety of accessibility issues and display the results in the panel, like this: You can see above there are a total of 50 issues, with 12 of them being critical. I'm going to click on the hyperlinked \u0026quot;12\u0026quot; to filter just the 12 critical issues:\nAlt-text issues OK now it's starting to get interesting. Our first problem is we have two images with no alt-text. This is a big no-no for accessibility, as screen readers rely on alt-text to describe images to users who can't see them.\nWhen I click on the \u0026quot;Images must have alt-text\u0026quot; link above, I see this:\nThere is great info here. You can see:\nthe flagged element location in the DOM actionable steps to address the issue highlighted element on the page (pink border) Form field issues Let's look at our other critical issues. The next issue is \u0026quot;form fields without labels\u0026quot;. This is a big deal because users who rely on screen readers need to know what each form field is for. If the form field is missing a label, or the label is not descriptive, it can be very confusing. Looks like we have 10 of those issues.\nIn our first example, we have the Name field, and just looking at it visually, it does appear to have a label. But the label is not connected to the input field in the DOM. This is a common issue with forms, and it's easy to fix.\nYou can see the problem element in the DOM, and the highlighted element on the page. The actionable steps tell you to connect the label to the input field.\nWrap-up So there you have it, some basics about how to use the axe DevTools extension to identify and fix accessibility issues.\nAlso, along the top of the page you wil see an \u0026quot;After\u0026quot; button that can show you a version of the page with all of the issues addressed, and 0 accessibility issues found in the scan.\nFinally, we looked at the 12 critical issues, but remember there are 50 total issues, and this is just for one page! Imagine how many issues you might find on a large site, especially one that has not been designed with accessibility in mind.\nIn addition to the browser extension, we can do accessibility testing in the CI pipeline with tools like Playwright. This approach ensures we catch these issues before they reach our codebase or get deployed to production. I'll cover that in my next blog post, so stay tuned!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/accessibility-testing/accessibility-testing-with-browser-plugin/","section":"blog","tags":["accessibility-testing"],"title":"Accessibility Testing with a Chrome Extension"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/testing/","section":"tags","tags":null,"title":"Testing"},{"body":"I've been blogging off-and-on for over 4 years. It's really helped me grow, and I usually recommend it to others as great way to keep growing skills.\nRecently I was giving peer review feedback to a fellow employee. As I was describing all of the reasons THEY might want to consider blogging, I realized I should create my own blog post that talks about the value of blogging.\nFocused learning, for me! It probably sounds weird, but a lot of the things I blog about are things I don't know much about when I start the blog.\nWhen I identify a skill or technology that's new to me and I want to learn more, many times I will write a blog as I work through my learning.\nFor example, let's say I want to get some expertise on using Postman for API testing, and I also want to demonstrate how to include those Postman tests in a CI pipeline.\nFor something like that, I will want to build a working prototype that demonstrates the key pieces of the tech I'm learning. As I build that prototype, I will also write a blog post that describes what I'm doing and why.\nWriting the blog as I work through the prototype reinforces my learning, and it forces me to feel sure I understand how things work. I don't want to share info with others that is incomplete or wrong.\nI'll also have screenshots and a repo I can share in the blog.\nFeel free to look at the blog I wrote for Postman. The blog actually started as a single post, but turned into a 2-part series. Once I got into it, I realized it was probably too big for one post, so I split it up.\nIntroducing myself to a future client I'm an IT consultant. That means I may be interviewing with new clients relatively frequently. I've found that making my blog posts available to potential clients is an AWESOME marketing tool!\nIf you have good content, and the person interviewing you has looked at it, that will get you a long way towards securing that next project or client or job.\nFor future reference Like many of us, I work with a lot of different technologies over time, and I need to be constantly growing my skills.\nAlong with all this learning comes a LOT of forgetting. The stuff that I was working with last year? That stuff that I knew like the back of my hand?? Well, it might be a distant memory to me now.\nIf I create a clear and concise blog about a topic as I work with it, I will have a future reference point that I can go back to as needed. Even if no one else reads my blog post, I can reference it.\nSince it's written in my voice, about a topic that I've struggled through, my memory will get refreshed a lot quicker than googling \u0026quot;help me with postman\u0026quot;.\nHelping others I guess this is an obvious benefit to blogging, and a bit less selfish than the previous ones I mentioned.\nWe've all been helped by so many nameless and faceless folks on our journey to improve our skills and marketability. Paying that back is a good thing and will make you feel good.\nSome suggestions These are just some random blogging suggestions that I thought of as I composed this.\nDon't compose in the blog provider's editor Avoid composing your blog directly in the blog platform's editor. Instead, consider composing your blog locally and pushing it to a repo. If you have a POC to support the blog, keep the blog with that code. From there you can just copy/paste it to the blog platform editor.\nIn the past I have run into issues where I've lost work with the blog platform editor, so I do everything locally. I just use VS Code for authoring markdown files and store them in GitHub.\nDon't get too hung up on visit and like counts I have yet to break the internet with any of my posts. I've had 1 or 2 that have done ok, but most have 10 likes or less. That's ok. Remember, blogging is for the benefit of the author also!\nI just remember that I'm good enough, I'm smart enough, and doggone it, people like me! Keep your post to a 5-minute read, or less Like everything in this post, this is just my opinion. I prefer shorter blog posts when reading them, so I try to do the same when writing them. If I find I have too much content, i can just split it into multiple posts.\nReference your repo and use screen shots This is pretty self-explanatory. If you are building something as you write you blog, include a link to your repo and don't forget to add some screen shots in the blog. Pictures are good!\nHmm, that Visual Basic 6 post may be too old... Blogs get stale, and they need to be updated or pruned. I have not done a good job with that, and I need to do better.\n(No, I don't really have a VB 6 post.)\nBlogging will give you more ideas Blogging will give you new ideas for more blogs, write that sh*t down so you don't forget!\nWrap-up So, there you go. That's a decent overview of why I blog. What do you think? Does this give you any ideas or motivation for blogging?\nWhat are some other good reasons to blog?\nFeel free to share your thoughts in the comments!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/why-i-blog/","section":"blog","tags":["testing"],"title":"Why I Blog"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/devops/","section":"tags","tags":null,"title":"Devops"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/postman/","section":"tags","tags":null,"title":"Postman"},{"body":"Introduction If you've been involved with application development or testing, you've probably used Postman to test API endpoints. In this post I'm going to create a sample endpoint, write Postman tests for the endpoint, and create a Github workflow to run those Postman tests whenever I push changes to the repo. The code we'll walk through can also be found in my repo.\nSo let's get started.\nThe app The point of this article is to walk through creating Postman tests and getting them running in a pipeline, so I'm not going to spend a lot of time with the sample app. To get an endpoint stood up quickly, I'm using an open source tool called JSON server.\nI'll avoid the temptation to tell you all the awesome things that JSON server can do for you. For now, I'll just say I can spin up a fully functional endpoint with just docker and a JSON file.\nThis endpoint is going to be pretty basic, and will provide basic CRUD operations for a list of albums.\nThe json file looks like this:\n1{ 2 \u0026#34;albums\u0026#34;: [ 3 { 4 \u0026#34;id\u0026#34;: 1, 5 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 6 \u0026#34;title\u0026#34;: \u0026#34;Please Please Me\u0026#34;, 7 \u0026#34;year\u0026#34;: \u0026#34;1963\u0026#34; 8 }, 9 { 10 \u0026#34;id\u0026#34;: 2, 11 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 12 \u0026#34;title\u0026#34;: \u0026#34;With the Beatles\u0026#34;, 13 \u0026#34;year\u0026#34;: \u0026#34;1963\u0026#34; 14 }, 15 { 16 \u0026#34;id\u0026#34;: 3, 17 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 18 \u0026#34;title\u0026#34;: \u0026#34;A Hard Day\u0026#39;s Night\u0026#34;, 19 \u0026#34;year\u0026#34;: \u0026#34;1964\u0026#34; 20 }, 21 { 22 \u0026#34;id\u0026#34;: 4, 23 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 24 \u0026#34;title\u0026#34;: \u0026#34;Beatles for Sale\u0026#34;, 25 \u0026#34;year\u0026#34;: \u0026#34;1964\u0026#34; 26 }, 27 { 28 \u0026#34;id\u0026#34;: 5, 29 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 30 \u0026#34;title\u0026#34;: \u0026#34;Help!\u0026#34;, 31 \u0026#34;year\u0026#34;: \u0026#34;1965\u0026#34; 32 }, 33 { 34 \u0026#34;id\u0026#34;: 6, 35 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 36 \u0026#34;title\u0026#34;: \u0026#34;Rubber Soul\u0026#34;, 37 \u0026#34;year\u0026#34;: \u0026#34;1965\u0026#34; 38 }, 39 { 40 \u0026#34;id\u0026#34;: 7, 41 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 42 \u0026#34;title\u0026#34;: \u0026#34;Revolver\u0026#34;, 43 \u0026#34;year\u0026#34;: \u0026#34;1966\u0026#34; 44 }, 45 { 46 \u0026#34;id\u0026#34;: 8, 47 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 48 \u0026#34;title\u0026#34;: \u0026#34;Sgt. Pepper\u0026#39;s Lonely Hearts Club Band\u0026#34;, 49 \u0026#34;year\u0026#34;: \u0026#34;1967\u0026#34; 50 }, 51 { 52 \u0026#34;id\u0026#34;: 9, 53 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 54 \u0026#34;title\u0026#34;: \u0026#34;The Beatles (White Album)\u0026#34;, 55 \u0026#34;year\u0026#34;: \u0026#34;1968\u0026#34; 56 }, 57 { 58 \u0026#34;id\u0026#34;: 10, 59 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 60 \u0026#34;title\u0026#34;: \u0026#34;Yellow Submarine\u0026#34;, 61 \u0026#34;year\u0026#34;: \u0026#34;1969\u0026#34; 62 }, 63 { 64 \u0026#34;id\u0026#34;: 11, 65 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 66 \u0026#34;title\u0026#34;: \u0026#34;Abbey Road\u0026#34;, 67 \u0026#34;year\u0026#34;: \u0026#34;1969\u0026#34; 68 }, 69 { 70 \u0026#34;id\u0026#34;: 12, 71 \u0026#34;artist\u0026#34;: \u0026#34;The Beatles\u0026#34;, 72 \u0026#34;title\u0026#34;: \u0026#34;Let It Be\u0026#34;, 73 \u0026#34;year\u0026#34;: \u0026#34;1970\u0026#34; 74 } 75 ] 76 } With just this file (and Docker) I can start my endpoint like this:\n1docker run -d -p 3005:80 -v /Users/denniswhalen/postman-workflow/db.json:/data/db.json clue/json-server You should now be about the hit that endpoint at http://localhost:3005/albums\nOnce we get some data back, we can create some Postman tests.\nThe Postman tests I'll assume you have familiarity with Postman. If you don't you can check out my previous post to get started, and of course there are many other fine resources available to get you started.\nMy first test is just going to do a GET request to http://localhost:3005/albums, verify that I get a response with status code of 200, and validate the schema of the response.\nThe test for my Postman request looks like this:\nI'm kind of glossing over this cool little Postman test that is validating the schema, but if you've never used Postman for schema validation it's definitely worth some more research. FYI, I used https://www.liquid-technologies.com/online-json-to-schema-converter to generate the expected schema.\nOnce I have my Postman test complete and have confirmed the test passes when it should and fails when it should, I will add it to my code repo. What repo you say? Well I guess I haven't mentioned that yet, but we're going to store our Postman collection is a Github repo. To do that you just need to use Postman to export the collection as a json file, and then add that file to your repo just like you would any other file.\nIf you push that change to Github, BOOM, you will see no workflow has run:\nWell of course. That's because we haven't created a workflow yet. We want a workflow that will run our Postman tests whenever code is pushed. If the Postman tests fails, we want the build to fail. Let's create the workflow now.\nThe workflow To allow Github to find the workflow file, the file needs to be in the ./.github/workflows folder. I'm naming my file workflow.yml, but the name doesn't matter. Only the file extension (.yml or .yaml) and file location matters.\nLet's take a look at the .yaml file for this workflow:\n1name: run Postman API tests 2on: 3 push: 4 branches: 5 - \u0026#39;main\u0026#39; 6 - \u0026#39;feature/**\u0026#39; 7 paths-ignore: 8 - \u0026#39;Postman Collections/**\u0026#39; 9 pull_request: 10 branches: [main] 11 workflow_dispatch: 12 13jobs: 14 build: 15 runs-on: ubuntu-latest 16 steps: 17 - name: checkout 18 uses: actions/checkout@v2 19 20 - name: start the app 21 run: | 22 docker run -d -p 3005:80 -v ${{ github.workspace }}/db.json:/data/db.json clue/json-server 23 24 - name: run postman tests 25 uses: matt-ball/newman-action@master 26 with: 27 collection: \u0026#34;Postman Collections/albums.json\u0026#34; This workflow is pretty basic and hopefully the steps of the \u0026quot;build\u0026quot; job make sense:\ncheckout - pull the code from the repo start the app - starts the json-server endpoint run postman tests - runs the postman tests in a container that has the Postman dependencies After adding the workflow.yml file to the repo and pushing, you he now see a workflow running un the actions tab in Github.\nHopefully you'll see something like this, indicating the test build was successful: If you open the \u0026quot;run postman steps\u0026quot; step you should see something like this, providing more detail about you Postmen test results: Wrap-up So there we go, we created an endpoint, wrote a cool Postman test, and created a repo with a workflow to run the tests. That wasn't too painful, right?\nFor our example, I mentioned exporting the Postman collection to json and adding it to the repo like you would with other files. There are a couple of other options for dealing with the Postman collection that I will cover in a future post.\nThe code I've talked about here can be found in my repo.\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/postman/running-postman-tests-in-a-pipeline/","section":"blog","tags":["postman","devops"],"title":"Running a Postman Collection in a CI Pipeline"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/cypress/","section":"tags","tags":null,"title":"Cypress"},{"body":"Introduction In my previous Cypress posts I've walked through how to get Cypress setup and running locally. To this point I haven't talked about how to run the tests in a CI pipeline. Well that all changes today!\nIn this post I'm going to show you how to easily get your tests running in Github Actions.\nGithub Workflow Github Workflow allows us to us define a workflow that that runs when certain triggering events occur, such as merging code to the repo. The workflow is defined with YAML, and is based on a documented formatting syntax.\nWhen the .yaml file is placed in the expected location and merged to the repo, it will be parsed by Github. If the file syntax is good, the workflow will be executed. You can view progress of the workflow execution via the Github UI.\nThe basic workflow I'm going to walk through will get you started, and I'll provide links help you learn more.\nSo let's do this!\nSetup For this post I'm assuming you have a bit of familiarity with Cypress. I have more detail in a previous Cypress post, so please take a look if you want that detail.\nI'm going to use the tests in the initial Cypress setup to get us started. To follow along at home:\ncreate a new folder for your project run npm init run npm install cypress --save-dev run npx cypress open run one of the tests through the Cypress runner to make sure everything looks good locally The workflow OK, so what do we want this workflow to do? Our workflow is going to be pretty basic. For every push to the main branch, the workflow will pull the code and run the Cypress tests on Firefox. If tests fail, the workflow fails. We also want to be able to trigger the workflow manually from the Github UI.\nLet's take a look at the .yaml file for this workflow:\n1name: run my vanilla Cypress tests 2on: 3 push: 4 branches: 5 - \u0026#39;main\u0026#39; 6 workflow_dispatch: 7 8jobs: 9 checkout-and-test: 10 runs-on: ubuntu-latest 11 steps: 12 - run: echo \u0026#34;🎉 The job was automatically triggered by a ${{ github.event_name }} event.\u0026#34; 13 14 - name: Check out repository code 15 uses: actions/checkout@v2 16 17 - name: run cypress tests with firefox 18 uses: cypress-io/github-action@v2 19 timeout-minutes: 10 20 with: 21 browser: firefox Hopefully the flow of this workflow is pretty self-explanatory. The key sections include:\nDefine the triggering events with on. We want our workflow triggered with a push to main or when manually triggered via the Github UI or APIs (workflow_dispatch): 1on: 2 push: 3 branches: 4 - \u0026#39;main\u0026#39; 5 workflow_dispatch: Checkout the code: 1- name: Check out repository code 2 uses: actions/checkout@v2 Run the tests on Firefox with the Cypress Github Action: 1- name: run cypress tests with firefox 2 uses: cypress-io/github-action@v2 3 timeout-minutes: 10 4 with: 5 browser: firefox For much more info regarding the Cypress Github Action, check out their detailed documentation. I'm just barely scratching the surface of its capabilities here.\nTo allow Github to find the workflow file, the file needs to be in the ./.github/workflows folder. I'm naming my file CI.yml, but the name doesn't matter. Only the file extension (.yml or .yaml) and file location matters.\nViewing workflow in Github Once your branch is pushed to Github, Github will parse the file and verify it conforms to the syntax specification. If you want to validate syntax before pushing, take a look at the Github Actions for VS Code plugin.\nTo view your workflow execution, go to your repo in the Github UI and click on the Actions tab. You should see your most recent workflow run at the top of the list: If the file syntax is not valid, the job will fail and you'll see an error. If the syntax is valid, you can click on the job and watch it as it runs. You should see messages very similar to want you see when running locally: To trigger the workflow via the UI, just click on your workflow in the left nav, and then click the Run workflow button:\nWrap-up So there we go, we now have our tests running with each push to the Github repo. Although the demo was focused on Github, you can accomplish the same with any of the major CI providers.\nI hope you learned something with this post, and can see just how easy it is to get your tests moved to a CI pipeline where they belong.\nAs always, let me know if you have any comments or questions!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/cypress/run-your-cypress-tests-in-a-github-workflow/","section":"blog","tags":["cypress","devops"],"title":"Run your Cypress Tests in a Github Workflow"},{"body":"Introduction In my previous post we installed Cypress, which also installed a number of Cypress sample tests. These sample tests use the Mocha syntax. In this post I'm going to talk about how to define your tests with feature files instead of Mocha. The examples will also leverage the page objects created in the previous post.\nBefore we get to the demo, I want to briefly discuss Gherkin and Cucumber.\nGherkin Gherkin and Cucumber are concepts that you'll hear when talking about Behavior Driven Design (BDD). BDD is not meant as a testing framework, but as a process that encourages communication and collaboration among business, development, and testing folks.\nThis collaboration occurs when we begin defining new requirements and stories, and uses concrete examples of user interaction and application responses. The \u0026quot;language\u0026quot; used to define these examples is called gherkin, but ultimately it's written in a manner that's understood by all, including the business. I wrote a blog post awhile back that talks more about best practices around Gherkin.\nFeature files are just the text files that contain the various concrete examples that were written in Gherkin.\nCucumber Cucumber comes into play when we want to build tests that automate the Gherkin stories. Cucumber is implemented for a number of frameworks and languages. My simplistic definition of Cucumber is \u0026quot;software that allows us to tie gherkin-based requirements to test automation code\u0026quot;.\nFor this demo I'll use cypress-cucumber-preprocessor, a Node package specifically developed to support feature files in Cypress.\nOk, it's time to write some code!\nSetup For this example I'm adding feature files to test the Cypress to-do sample app, and I will be converting one of the Cypress spec file tests to a feature file.\nIf you want to follow along with this example, take a look at the Setup section of my previous post, Using Page Object in Cypress\nAdding Cucumber support to the project is a relatively easy task with the help of cypress-cucumber-preprocessor. Here are the steps:\nInstall cypress-cucumber-preprocessor 1npm install --save-dev cypress-cucumber-preprocessor Add cypress-cucumber-preprocessor to the integration/plugins/index.js 1const cucumber = require(\u0026#39;cypress-cucumber-preprocessor\u0026#39;).default 2module.exports = (on, config) =\u0026gt; { 3 on(\u0026#39;file:preprocessor\u0026#39;, cucumber()) 4} Update cypress.json so Cypress knows that tests are contained in .feature files. I added feature files, and also decided to ignore the original sample tests that were installed with Cypress. 1{ 2 \u0026#34;testFiles\u0026#34;: \u0026#34;**/*.{feature,features,spec.js}\u0026#34;, 3 \u0026#34;ignoreTestFiles\u0026#34;: [ 4 \u0026#34;**/1-getting-started/*.js\u0026#34;, 5 \u0026#34;**/2-advanced-examples/*.js\u0026#34; 6 ] 7} Create a feature file I am going to be referencing some files that are found in my github repo, specifically in the 04-cucumber-examples branch. It might help to reference that repo as you read through this.\nLet's take a look a the first first test in 3-page-object-examples/todo-po-class-spec.js:\n1beforeEach(() =\u0026gt; { 2 todoPage.navigateToHome(); 3 }) 4 5 it(\u0026#39;displays two todo items by default\u0026#39;, () =\u0026gt; { 6 todoPage.validateTodoCount(2) 7 todoPage.validateTodoText(1, \u0026#39;Pay electric bill\u0026#39;) 8 todoPage.validateTodoText(2, \u0026#39;Walk the dog\u0026#39;) 9 }) As you can see, we're navigating to the home page, then validating the ToDo count and ToDo content. In gherkin, it might look something like this:\n1Feature: a sample feature to practice my testing 2 3Scenario: displays two todo items by default 4 When I open the to-do page 5 Then 2 to-do items are displayed 6 And to-do item 1 is \u0026#34;Pay electric bill\u0026#34; 7 And to-do item 2 is \u0026#34;Walk the dog\u0026#34; Note I said \u0026quot;it might look something like this\u0026quot;. Other than some keywords, the scenario should be written in a common language that makes sense to all. I don't like those 2 hardcoded ToDo's, but I am going to try and ignore that for now since the point of this post is just to show you how to use feature files with Cypress.\nNow I just need to create a .feature file and put it somewhere in the integration folder. You can paste the above scenario into a new file namedintegration/Sample.feature\nCreate a step definition file So now we have a feature file, and we already had a page object, but we need to tie them together. That's where the step file comes in. Each step in your Gherkin scenario will need to match to a step in a step file. The step file will then call the appropriate page object.\nThe location of your step files is configurable based on a number of factors. Take a look at the doco for cypress-cucumber-preprocessor to get more info. I have things setup to put my feature files in the support/step_definitions folder.\nLet's focus on the first step of the scenario:\n1When I open the to-do page To create the step file for this step, I'm just going to create file support/step_definitions/to-do-steps.js and paste the following:\n1import { TodoPage } from \u0026#34;../../page-objects/todo-page\u0026#34; 2import { Given, When, Then } from \u0026#34;cypress-cucumber-preprocessor/steps\u0026#34;; 3 4const todoPage = new TodoPage() 5 6When(\u0026#39;I open the to-do page\u0026#39;, () =\u0026gt; { 7 todoPage.navigateToHome(); 8}) Hopefully it's pretty clear what's gong on here. We are importing the todo-page object and step syntax from cypress-cucumber-preprocessor. The step in the step file matches the step in the gherkin, so it will call navigateToHome() in the page object.\nRunning our cucumber tests So we implemented our feature file and steps file, let's run the feature file:\n1npx cypress run --spec \u0026#34;cypress/integration/**/Sample.feature\u0026#34; If things work as expect, you should see error:\n1Error: Step implementation missing for: 2 to-do items are displayed That's what you want to see. We have only implemented the first step of our test scenario. To get a green test and verify the first step is working, let's comment out the steps we haven't implemented and run again:\n1Feature: a sample feature to practice my testing 2 3Scenario: displays two todo items by default 4 When I open the to-do page 5# Then 2 to-do items are displayed 6# And to-do item 1 is \u0026#34;Pay electric bill\u0026#34; 7# And to-do item 2 is \u0026#34;Walk the dog\u0026#34; Hopefully when you run npx cypress run --spec \u0026quot;cypress/integration/**/sample.feature\u0026quot; you see a green test. Once it's green you can move ahead more quickly to implement steps for the others, following the same pattern.\nOf course the first one is always the most difficult. If things are still not green, you will probably need to resort to reading the error messages. You can also look at my github repo to find differences.\nWrap-up Well I hope you found some value in this post. Writing this post took longer than I expected, and I still feel like I have not gone into enough detail. If you have any questions or suggestions, please feel free to leave comments or contact me.\nThanks and GO BENGALS!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/cypress/using-gherkin-with-your-cypress-tests/","section":"blog","tags":["cypress"],"title":"Using Gherkin with your Cypress Tests"},{"body":"Introduction Page Objects is a pattern in test automation that allow the automation engineer to encapsulate the data and methods used to support automation of a page. Typically each page of the application will have an automation class that contains data, methods, and locators needed for automation of that page.\nIn this post I will take a look at a sample Cypress script that does not use Page Objects, and walk you through the steps of converting to the Page Object pattern.\nAdvantages So why do we want to use Page Objects? Some advantages of this pattern include:\nseparating the implementation complexity of automation from the more business-focused details of the test script steps providing a single location for automation details, so if something like a page locator changes in the application, we only need to change it in one place in the automation code But enough talking, let's get started. Hopefully we can see these advantage in action!\nSetup To make this easy to follow along, I'm going to be starting with the sample test scripts that are included when you install Cypress. Let's do that:\nmake sure you have node installed create a new folder for our example and from the command line run: npm init -y install cypress with: npm install cypress --save-dev start the test runner with: npx cypress open After starting the test runner, you'll see a cypress folder structure like this:\nIn the 1-getting-started folder you'll see todo.spec.js. This is a sample test that comes with Cypress and demonstrates some basic functionality using https://example.cypress.io/todo as the application under test.\nTo complete the setup, let's just make sure this test runs properly. Run the test from the command line with npx cypress run --spec cypress/integration/1-getting-started/todo.spec.js. If things go as planned, you should see messages that 6 tests ran and something like this towards the end:\nThere we go, setup complete!\nPlanning our page class - round 1 Let's take a closer look at the todo.spec.js file. The beforeEach() hook is not all that interesting, but let's start with that since it's pretty easy to understand.\n1beforeEach(() =\u0026gt; { 2 cy.visit(\u0026#39;https://example.cypress.io/todo\u0026#39;) 3 }) Instead of having the cy command and the destination URL in the spec, let's create a page object method to deal with all that.\nCreating the page object class The class we're going to create is just a JavaScript class. Create a new folder in the integration folder and name it something like page-objects. Create a new file names todo-page.js in the page-objects folder.\nAdd the following to the new class file:\n1export class TodoPage { 2 3 navigateToHome() { 4 cy 5 .visit(\u0026#39;https://example.cypress.io/todo\u0026#39;) 6 } 7} Update the spec to use the page object We have a new method named navigateToHome() that we're going to call from our spec file. To make that happen, we just need to make a few updates to the todo.spec.js file.\nadd the following import: 1import { TodoPage } from \u0026#34;../page-objects/todo-page\u0026#34; instantiate the ToDoPage object prior to the beforeEach hook: 1const todoPage = new TodoPage() update the beforeEach hook to call the page object method: 1 beforeEach(() =\u0026gt; { 2 todoPage.navigateToHome(); 3 }) Those are the changes. Run the test again and make sure it's still green:\nnpx cypress run --spec cypress/integration/1-getting-started/todo.spec.js\nAnd that's about it! We've created a page object class, and used it from our test spec.\nPlanning our page class - round 2 There are other examples in the spec that might benefit more from Page Objects. Take a look at the first line of the first test:\n1cy.get(\u0026#39;.todo-list li\u0026#39;).should(\u0026#39;have.length\u0026#39;, 2) This step is verifying that our app has two to-do items. The Cypress interaction detail is right there in the spec, with a lot of locator and cypress detail that could be moved to the page object.\nAlso what about other tests that follow this pattern? The locator for the to-do list items will be scattered throughout our tests. What if that locator changes in the future? Yup, lots of updates and all the work and risk that comes with that.\nI want to move all that detail into one place, the page object. Our spec will become more clear, with a statement such as:\n1todoPage.validateTodoCount(2) See how clear that is? It's so clear I am not even going to explain it. After our first example, it's probably pretty clear how to do this in the page object class. Let's take a look.\nUpdating the page object class Since we already had the implementation details in the spec, we can just copy/paste/tweak that for our new Page Object method:\n1 validateTodoCount(expectedLength) { 2 cy 3 .get(\u0026#39;.todo-list li\u0026#39;) 4 .should(\u0026#39;have.length\u0026#39;, expectedLength) 5 } So with those changes to the spec and page object class, run the test again and make sure you are still seeing green tests.\nCool, right? And the great thing? Any test that needs to validate the to-do count in the future can just use this method without worrying about locators or Cypress.\nWrap-up So with those basic examples I hope you can see the value of encapsulating your automation code into Page Objects. Feel free to experiment on your own and convert ALL the tests in that that spec to use Page Objects. It's a great exercise.\nI should also point out that since our Page Object class does not have any instance data, we could have just used functions instead of the class. To learn about this option and much more, I highly recommend you look at Cypress courses Introduction to Cypress by Gil Tayer and Advanced Cypress by Filip Hric, available for free from Test Automation University. Much of my initial Cypress interest and learning came from these 2 courses.\nAnd finally, I would be remiss if I did not mention the article by the Gleb Bahmutov Stop using Page Objects and Start using App Actions where he explores some alternatives to Page Objects.\nThat's all for now, thanks!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/cypress/using-page-objects-in-cypress/","section":"blog","tags":["cypress"],"title":"Using Page Object in Cypress"},{"body":"This post is a little different from my typical post. It's really just a question.\nI'd love to get some feedback, suggestions, and discussion based on YOUR experience working with DEV and QA on agile projects.\nSo what is the question? I've been involved with my fair share of both waterfall and agile projects. I've also been involved with a lot of \u0026quot;wagile\u0026quot; projects, a combination of waterfall and agile.\nOn these wagile projects the team is working in sprints, is agile-ish, but DEV and QA teams are siloed, working on different goals within the same sprint. For example, DEVs complete coding and (hopefully) unit testing for a user story in sprint 1, and QA does manual testing and and builds automated tests for that user story in sprint 2.\nSome drawbacks I see here are:\nNo fast feedback By the time QA finds a defect it could be 2 weeks old. Does the developer remember all the detail about a task they worked on 2 weeks ago? Maybe not. Quick feedback allows the developer to keep focus on the task at hand.\nNo communication between DEV and QA Well, there's some communication, but it's mostly bad news, such as \u0026quot;it appears your code is broken and I have formally documented it for you\u0026quot;. Ideally DEV and QA would be working as one TEAM, with the same goals, in the same sprint.\nFinger pointing Because DEV and QA really don't communicate until there's a defect, I usually see finger pointing, blame, and a general salty attitude between the 2 teams. QA doesn't understand how the broken code could have ever been considered \u0026quot;done\u0026quot;, and DEV is moving on to new work that they've committed to for the sprint. The 2 teams have different goals within the same sprint.\nStory pointing and sprint planning becomes weird For these types of projects, the DEV and QA tasks usually end up getting sized and planned separately. Ideally the stories would be sized by the TEAM, but with 2 teams working on different goals in the same sprint, sizing usually gets split also.\nQA is always \u0026quot;behind\u0026quot; With this strategy, QA is always playing catch-up, and is usually considered a bottleneck to moving more quickly. DEVs really have no incentive to help QA with issues or testing, as they have moved on to new user stories for the sprint.\nThe codebase is not ready to deploy Because you have code in your codebase that has not been thoroughly tested, you have code in your codebase that is not ready to deploy.\nWhat's the solution? So those are a few things I've seen in the past. IMHO the solution to most of this is to completely get rid of the concept of a DEV team and a QA team. The agile team needs to be ONE TEAM that works together to complete development and testing of the user stories in the same sprint.\nSome folks may have specialties within the team, but the whole team has the same goal: meet sprint commitments by completing development and testing of user stories within the same sprint. Everyone on the team pitches in to ensure the TEAM meets their commitments.\nWhat are YOUR thoughts? I'd love to hear some feedback from you. Do you see similar challenges with your projects? How do your teams deal with them? How can we help move a team from wagile to agile?\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/one-team-dev-and-qa/","section":"blog","tags":["shift left"],"title":"One Team, Dev and QA"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/shift-left/","section":"tags","tags":null,"title":"Shift Left"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/test-data/","section":"tags","tags":null,"title":"Test Data"},{"body":"Intro When building a QA test automation framework, it's critical to define the strategy for dealing with test data. For example, what’s wrong with this scenario?\n1Given the user successfully logs into registration system 2When the user searches for course number “ALG-4316” 3Then the course is displayed 4And the course has title of “Intermediate Algebra” 5And the course has credit hours of “4” Clearly the scenario contains hard-coded test data: course number, course name, and credit hours.\nWhen first building test automation it's easy to start like this. We're testing in the dev environment, we know this course exists in the database, so let's start building some automated tests! Although it's easy to get started like this, it can quickly become impossible to sustain.\nLet's take a look at some alternatives for dealing with test data.\nOption 1 - Hard-coded test data Overview The architecture behind the hard-coded test data option is not too difficult to understand. With this option, the test data is defined within the test process and stored in data files or hard-coded directly in the test.\nPros:\nStart building and completing tests quickly Cons:\nLong term test maintainability and brittleness Issues running tests in different environments/databases Could require a database restore strategy prior to running the test When appropriate? Although all tests may contain some hard-coded test data, it’s really only appropriate for [1] static data that will not change across environments, for [2] data that is not relevant to the test, or for [3] test setups where you can always restore the database to a known state before testing.\nOPTION 2 - Real time test data extract Let’s try the gherkin again and remove the hard-coded test data:\n1Given the user successfully logs into registration system 2When the user searches for an active course 3Then the course is displayed 4And the course title is correctly displayed 5And the course credit hours are correctly displayed With this option we are no longer defining the test data in the test. The automation code that’s executed behind this test will be responsible for identifying the necessary test data and expected results. But how?\nOverview With real time test data extract, the test process will extract the necessary test data from the application as we execute the test. So instead of hard-coding the course number, we can first extract a valid course number and associated course detail from the application, and then use that data in our test.\nAs shown in the diagram below, the data can be extracted via an existing application API or by reading it directly from the application database.\nPros:\nTests are not reliant on specific hard-coded test data Tests will work across multiple test environments Cons:\nNeed a reliable mechanism to read from the application (API or DB) Assumes the database already contains data you need Could have issues with concurrent tests grabbing same test data When appropriate? This option is appropriate when [1] the necessary test data exists in the database, [2] there are no concerns with concurrent tests potentially using the same data, and [3] we have read access to the database, either through API or direct database access.\nOPTION 3 - Real time seeding Overview With real time test data seeding, the test process will inject the necessary test data into the application as we execute the test. So instead of testing with an existing course number, we can first inject a valid course number and associated detail into the application, and then use that data in our test.\nAs shown in the diagram below, the data can be injected via an existing application API or by inserting it directly from the application database.\nPros:\nTests are not reliant on specific hard-coded test data Tests will work across multiple test environments No data sharing issues with concurrent tests Cons:\nNeed a mechanism to write to database (API or direct) Seeding data may be more complex that just extracting existing data When appropriate? This option is appropriate when [1] the necessary test data may not already exist in the database, [2] tests will run concurrently and can’t share the same data, and [3] we have write access to the database, either through API or direct database access.\nOPTION 4 - Decoupled extract The options describe above may work well for functional testing, but what about performance testing? With performance testing we’ll simulate multiple concurrent users to verify the application can handle the required workload and respond within the expected response times.\nInteracting with the application to acquire test data during a performance test will just apply more load to the application. This load is not representative of a real-world scenario, and would likely skew the test results and server performance statistics.\nTo get around this issue we can decouple the test data creation process from the testing process.\nAs represented above, the necessary test data is injected into the application prior to the start of the performance testing. In addition to injecting the test data, we’ll also write that test data to a data repository on a server that is separate from the application.\nWhen the performance test starts, it will read test data from that test data repository. This strategy will allow the performance tests to be driven by test data without seeding or extracting it during the test.\nAlthough this process is probably the most complex method for dealing with test data, it does provide an additional bonus; the test data generated for performance testing can also be consumed by the functional UI and API tests, as depicted above.\nConclusion As you define the test data strategy for your project, there is clearly not one correct way to do it. In fact, you will likely use various options on the same project, depending on the needs of specific tests.\nRegardless of your strategy, be sure to think about it up front, and refine it as you learn more about what works and what doesn’t.\n","link":"https://dennis-whalen-dot-com.vercel.app/test-data-strategies/","section":"","tags":["test data"],"title":"Test Data Strategies for a Test Automation Framework"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/tags/api/","section":"tags","tags":null,"title":"Api"},{"body":"In this post we'll talk about using Cypress to run API tests. But what is Cypress?\nCypress is an open sourced JavaScript-based test automation framework that is typically used for testing web applications that leverage modern JavaScript frameworks.\nIf you're just getting started with Cypress, you might want to check out my previous post to get a general understanding of Cypress and how to get it running in your environment. In that post I covered how to install Cypress locally and get some UI tests running. In this article I am going to continue adding to that project for the API testing.\nSetting up JSON-Server Before we we can test API endpoints, we need some API endpoints to test. For that I am going to quickly setup some endpoints on my machine that I can test using JSON Server.\nJSON Server is an open-sourced Node module you can use to quickly setup test endpoints for mocking and testing.\nAdd dependencies for json-server and faker To get our endpoint running locally we need to add json-server and faker to our package.json file. Faker will be used to generate some random data. Update your dev-dependencies section to look something like this:\n1\u0026#34;devDependencies\u0026#34;: { 2 \u0026#34;cypress\u0026#34;: \u0026#34;^6.9.1\u0026#34;, 3 \u0026#34;faker\u0026#34;: \u0026#34;^4.1.0\u0026#34;, 4 \u0026#34;json-server\u0026#34;: \u0026#34;^0.15.0\u0026#34; 5 } Run npm install to load install the newly added modules.\nAdd a process to load data into our endpoint JSON Server needs some data to serve through our endpoint. To make that happen, create a file named employees.js and paste the following into it:\n1var faker = require(\u0026#39;faker\u0026#39;) 2function generateEmployees () { 3 var employees = [] 4 for (var id = 0; id \u0026lt; 50; id++) { 5 var firstName = faker.name.firstName() 6 var lastName = faker.name.lastName() 7 var email = faker.internet.email() 8 employees.push({ 9 \u0026#34;id\u0026#34;: id, 10 \u0026#34;first_name\u0026#34;: firstName, 11 \u0026#34;last_name\u0026#34;: lastName, 12 \u0026#34;email\u0026#34;: email 13 }) 14 } 15 return { \u0026#34;employees\u0026#34;: employees } 16} 17module.exports = generateEmployees This function will just return an array of 50 employees with random names and emails.\nStart json-server From that command line run json-server employees.js. This will start json-server and the endpoint will return the randomly generated employees. If things go as expected, you should see something like this: You should now have a API endpoint at http://localhost:3000/employees, and can view that data in your browser: In addition to providing an endpoint to retrieve your data, JSON Server allows you to make POST, PUT, PATCH or DELETE requests, making it an ideal solution for mocking endpoints.\nWith JSON Server you can start building test for API endpoints before the developer implements the endpoint. You can easily mock the expected responses and once the API is implemented, your tests can point to that implementation instead of JSON Server. Development and API test automation can happen in parallel.\nNow let's create some tests!\nYour first API tests To get started we're going to create a test that performs a GET request to our /employees endpoint and verifies the response is JSON, the status code is 200, and the response contains 50 employees.\nCreate file named employeeTests.js with the following content:\n1describe(\u0026#39;employees API\u0026#39;, () =\u0026gt; { 2 it(\u0026#39;verify request returns JSON\u0026#39;, () =\u0026gt; { 3 cy.request(\u0026#39;http://localhost:3000/employees\u0026#39;).its(\u0026#39;headers\u0026#39;).its(\u0026#39;content-type\u0026#39;).should(\u0026#39;include\u0026#39;, \u0026#39;application/json\u0026#39;) 4 }) 5 6 it(\u0026#39;verify the request returns the correct status code\u0026#39;, () =\u0026gt; { 7 cy.request(\u0026#39;http://localhost:3000/employees\u0026#39;).its(\u0026#39;status\u0026#39;).should(\u0026#39;be.equal\u0026#39;, 200) 8 }) 9 10 it(\u0026#39;verify the request returns 50 items\u0026#39;, () =\u0026gt; { 11 cy.request(\u0026#39;http://localhost:3000/employees\u0026#39;).its(\u0026#39;body\u0026#39;).should(\u0026#39;have.length\u0026#39;, 50) 12 }) 13}) There are 3 tests here that will do our preliminary validation. Lets run the tests!\nStart the test runner You can start the test runner with the command: ./node_modules/.bin/cypress open\nIf you need help with this, refer back to my previous post to get you started.\nRun the tests Once the Cypress Runner starts you should see your employeeTests.js file. Be sure you've started json-server and then run the test by clicking the file in the Cypress Test Runner. In no time you will see your results: Make sure your test fails I usually don't trust my tests until I see them fail. I am going to edit the employeeTests.js file to change the expected status code to 201 just so I can be sure these tests are working. Rerun the test and you should see something like this: A couple things of note here:\nthe Test Runner allows me to rerun tests without reloading the runner, and it is immediately aware of any changes I make to the tests. if you expand the failed test you can get some more useful info about the failure: More API tests So now we have a test that verifies we can get our list of employees. Let's create a test that verifies we can add and delete an employee.\nCreate file named addDeleteEmployeeTest.js with the following content:\n1let newId; 2 3describe(\u0026#39;employees API\u0026#39;, () =\u0026gt; { 4 5 it(\u0026#39;Add a new item\u0026#39;, () =\u0026gt; { 6 cy.request(\u0026#39;POST\u0026#39;, \u0026#39;http://localhost:3000/employees\u0026#39;,{ first_name: \u0026#39;New\u0026#39;, last_name: \u0026#39;Dude\u0026#39;, email: \u0026#34;new_dude@googling.com\u0026#34; }) 7 .its(\u0026#39;body\u0026#39;).then((body) =\u0026gt; { 8 newId = body.id; 9 }) 10 }) 11 12 it(\u0026#39;Verify the new item exists exists\u0026#39;, () =\u0026gt; { 13 cy.request(\u0026#39;http://localhost:3000/employees/\u0026#39; + newId).then((response) =\u0026gt; { 14 expect(response.status).to.eq(200) 15 expect(response.body).to.have.property(\u0026#39;first_name\u0026#39;, \u0026#39;New\u0026#39;) 16 expect(response.body).to.have.property(\u0026#39;last_name\u0026#39;, \u0026#39;Dude\u0026#39;) 17 expect(response.body).to.have.property(\u0026#39;email\u0026#39;, \u0026#39;new_dude@googling.com\u0026#39;) 18 }) 19 }) 20 21 it(\u0026#39;Delete the newly added item\u0026#39;, () =\u0026gt; { 22 cy.request(\u0026#39;DELETE\u0026#39;, \u0026#39;http://localhost:3000/employees/\u0026#39; + newId).then((response) =\u0026gt; { 23 expect(response.status).to.eq(200) 24 }) 25 }) 26 27 it(\u0026#39;Verify the item was deleted\u0026#39;, () =\u0026gt; { 28 cy.request({url: \u0026#39;http://localhost:3000/employees/\u0026#39; + newId, failOnStatusCode: false}).then((response) =\u0026gt; { 29 expect(response.status).to.eq(404) 30 }) 31 }) 32}) Hopefully these tests are pretty self explanatory. With these new tests we are validating we can add and delete a new employee. Go ahead and run addDeleteEmployeeTest.js from the Cypress runner. You should see something like this: These results show we are able to add and delete a new employee via our API endpoint.\nRunning a test from the command line As mentioned in my previous post, we ultimately we want these tests to run in a CI pipeline, which means we need to be able to run them from the command line. Running from the command line is easy enough:\n1./node_modules/.bin/cypress run --spec cypress/integration/examples/addDeleteEmployeeTest.js You should see something like this: What's Next? In my first 2 posts I've shown you how to get Cypress running locally and get some UI and API test running. I feel like we've covered a lot, but we've really only scratched the surface. The Cypress site has a lot more detailed documentation and examples of the power of Cypress.\nIn future posts I want to look into what we need to do to move these tests from our local environment into a CI pipeline, and I also want to look at some reporting options.\nIf there are any questions, or ideas for future posts, please let me know in the comments.\nUntil then, stay tuned!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/cypress/api-testing-with-cypress/","section":"blog","tags":["cypress","api"],"title":"API Testing With Cypress"},{"body":"Ever seen a Gherkin scenario that looks like this?\n1Scenario: Prime customers shipping is free for Prime items 2 3Given I access URL \u0026#34;https://www.amazon.com\u0026#34; 4And I click the \u0026#34;Hello, Sign in\u0026#34; link 5And the \u0026#34;Sign-In\u0026#34; page is displayed 6When I enter email address \u0026#34;someprimeemailaddress@thisisfake.com\u0026#34; 7And I click the \u0026#34;Continue\u0026#34; button 8Then the \u0026#34;Password\u0026#34; page is displayed 9When I enter email address \u0026#34;donotputpasswordsingherkin\u0026#34; 10And I click the \u0026#34;Continue\u0026#34; button 11Then I am logged in 12And the \u0026#34;Home\u0026#34; page is displayed 13When I enter \u0026#34;Crocs that will make me look cool\u0026#34; in the search textbox 14And I click the hourglass image 15And the search results are displayed 16And \u0026#34;Crocs shoes\u0026#34; are displayed on the search results page 17And I click on the first item in the search results 18And the \u0026#34;Item Detail\u0026#34; page for the selected item is displayed 19And I select \u0026#34;13 Women/11 Men\u0026#34; from the size combo box 20And the \u0026#34;Inspired By\u0026#34; page is displayed 21When I click the \u0026#34;Proceed to Checkout\u0026#34; button 22Then the \u0026#34;Checkout\u0026#34; page is displayed 23And there is 1 item in the order 24And The Shipping \u0026amp; Handling cost for the order is \u0026#34;$0.00\u0026#34; 25And I click the \u0026#34;Place your Order\u0026#34; button 26And the \u0026#34;Order Placed\u0026#34; page is displayed Besides searching for \u0026quot;Crocs that will make me look cool\u0026quot;, what is wrong with this scenario?\nIn my role, I focus on QA Test Automation and I work with clients to build automated tests. Many times those tests are driven by Gherkin scenarios, and I have seen scenarios like the one above.\nI have a couple Gherkin rules that I try to encourage that should help out with this scenario.\nRule 1 - use declarative syntax when writing scenarios One of the goals of Behavior Driven Design (BDD) and Gherkin in general is to provide a mechanism to facilitate communication between the business stakeholders and the development team. The goal of these scenarios is to describe business rules in the domain language understood by the business.\nThe scenario above uses an imperative style of communication. In this example we are using the scenario as a way to describe HOW to complete a task, instead of WHAT the user does. This can lead to a long list of tasks, without clear direction on the business rule being described. What is the rule being described here? Looks like lots of things! (see rule 2)\nAnother drawback is that it's very painful to read and verify. Imagine a 2-hour meeting with QA, developers, and stakeholders where you review 30 scenarios like this. They are boring, hard to follow, and easy to ignore.\nWith declarative language we can describe WHAT the user does, without describing HOW to do it. If you find yourself referring to buttons and text boxes in your scenarios, consider other options. In fact, try to imagine that you are describing business rules without referring to an application at all. For example, we could replace the first 11 lines with a single line:\n1Given a Prime customer is authenticated Rule 2 - only one When/Then combination per scenario Looking at the scenario above it's clear we would are describing an end-to-end scenario instead of describing one behavior. With BDD you want one scenario to describe a single behavior, and that means you want a single When/Then combination. The \u0026quot;When\u0026quot; describes the user interaction and the \u0026quot;Then\u0026quot; describes the expected results. So, the question is, what behavior are we describing with this scenario?\nIt's hard to say. There are plenty of \u0026quot;Whens\u0026quot; and \u0026quot;Thens\u0026quot;, but looking at the scenario description I see \u0026quot;Prime customers shipping is free for Prime items\u0026quot;. Let's assume that's the business rule we want to describe. So let's try this:\n1Scenario: Prime customers shipping is free for Prime items 2 3Given a Prime customer is authenticated 4When the authenticated customer enters checkout with a single Prime item 5Then the shipping is free How's that? It's certainly much easier to read, it's very clear what the business rule is, and it provides a concrete example for validating it works as expected.\nIn addition, focusing on the business rule allows us to see other things we need.\nWhat if the item is not Prime eligible? We need a scenario for that. What if there are multiple Prime items? We need a scenario for that. What if the order is a combination of Prime and non-Prime items? We need a scenario for that. What if the customer is not a Prime customer? We need a scenario for that. Wrap-up Rules and standards for writing Gherkin can actually be a pretty controversial topic. If you are starting a new project, NOW is the time to reach agreement on how you are going to use BDD and Gherkin on your project.\nIf the project is well underway and has lots of imperative/procedural scenarios in place, backed by an automation framework to automate those steps, it will be more challenging to make a change.\nIn that situation, baby steps may allow you to agree write new scenarios with a declarative syntax. For test automation you could abstract the multiple procedural steps into a single declarative step and continue to use your automation steps under the hood.\nAnd if you're still not sure, imagine how much more pleasant those scenario review meetings are going to be!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/wow-thats-some-boring-gherkin/","section":"blog","tags":["testing"],"title":"Wow, That's some Boring Gherkin!"},{"body":"IThis post is a sequel to my first Gatling post. I know what you're thinking, isn't the sequel always lame? Well yes, that is usually true. Here's hoping I can reverse that trend! With this post we'll get a little deeper into the nitty gritty of Gatling.\nMy previous post started with a general overview of performance testing, and then dove into using the Gatling Recorder to build a Gatling script. In those examples we used the Gatling bundle, but there is another way.\nThe Gatling bundle is a standalone tool, and is not really the tool you'll use in when leveraging Gatling in an enterprise environment. Gatling can also be used with a build tool such as Maven, sbt, or Gradle. Let's bootstrap a new Maven project by using the Maven archetype.\nBootstrapping a new project for Gatling You can use your IDE to create a new project using an archetype, but let's just do it from the command line:\n1mvn archetype:generate -DarchetypeGroupId=io.gatling.highcharts -DarchetypeArtifactId=gatling-highcharts-maven-archetype You'll be prompted to enter a groupId, artifactId, version, and package. Once you confirm your selections, a new project will be created. After that you may also need to setup your Scala SDK. And you may also need to mark your src/test folder as Test Sources Root. Take a look at your pom.xml file. There you will see the dependencies for Gatling. Running the recorder In my first post I opened the recorder from the bin folder of the Gatling bundle. With the Maven project you can open it by running src/test/resources/Recorder.scala. After a few seconds you should see the recorder utility, which can be used to record scripts. Once opened, the Recorder functionality is the same as I described previously. The Recorder utility should look like this: I just showed you how to run the Recorder from the Maven project. So now it's time to use the Recorder right? Sorry no, plot twist!\nThe Recorder can be used to generate a script that will run, but of course it's important to understand what that generated code is doing. IMHO the Recorder alone is not sufficient to get you far with Gatling. Instead of using the Recorder, let's create a script from scratch! (See the previous post for a demo of the Recorder.)\nWriting your first script from scratch To create the script I will be using the Community Edition of IntelliJ IDEA. If you don't already have it, you can download and install. Once installed, open the project you created previously.\nCreate a new package in the src/test/scala folder named scripts.\nCreate a new Scala class in that package named HelloGatling and open it for editing. This is going to be our script.\nThere are 2 imports that are required for all Gatling scripts, so let's start by adding those to the class:\n1import io.gatling.core.Predef._ 2import io.gatling.http.Predef._ Your class will need to extend the Gatling Simulation class to make it into a Gatling script:\n1class HelloGatling extends Simulation { 2 3} Now we're ready to start working on our class. Let's start with the http config, which defines the base URL and will sets up our header to send json formatted data.\n1val httpConfig = http 2 .baseUrl(\u0026#34;http://computer-database.gatling.io\u0026#34;) 3 .header(\u0026#34;Accept\u0026#34;, \u0026#34;application/json\u0026#34;) Now we're ready to build the scenario, which will describe all the interactions our script makes with the application. For this demo our script will access the Gatling sample web site. Let's first access the home page:\n1val myScenario = scenario(\u0026#34;Add a new computer scenario\u0026#34;) 2 .exec( 3 http(\u0026#34;load the Home Page\u0026#34;) 4 .get(\u0026#34;/\u0026#34;) 5 ) With this code we have created a scenario with a single step. We give text descriptions for the scenario and for the step. These descriptions will show up on the Gatling report. The get statement accesses the Home page.\nNow we can add the remaining steps that [1]access the Add New Computer page and then [2]post the new computer. The complete scenario looks like this:\n1val myScenario = scenario(\u0026#34;Add a new computer scenario\u0026#34;) 2 .exec( 3 http(\u0026#34;load the Home Page\u0026#34;) 4 .get(\u0026#34;/\u0026#34;) 5 ) 6 .pause(5) 7 8 .exec( 9 http(\u0026#34;get the Add New Computer page\u0026#34;) 10 .get(\u0026#34;/computers/new\u0026#34;) 11 ) 12 .pause(5,10) 13 14 .exec( 15 http(\u0026#34;Post the new computer\u0026#34;) 16 .post(\u0026#34;/computers\u0026#34;) 17 .formParam(\u0026#34;name\u0026#34;, \u0026#34;Ionic Defibulizer\u0026#34;) 18 .formParam(\u0026#34;introduced\u0026#34;, \u0026#34;2020-01-01\u0026#34;) 19 .formParam(\u0026#34;discontinued\u0026#34;, \u0026#34;2020-06-30\u0026#34;) 20 .formParam(\u0026#34;company\u0026#34;, \u0026#34;37\u0026#34;) 21 ) In addition to the 2 new steps, we also have also added a pause between them. pause(5) will pause 5 seconds and pause(5,10) will pause a random amount of time between 5 and 10 seconds. Pause time is there to simulate how a user will interact with the page. Realistic pause time is critical to building realistic scenarios.\nOK we have our scenario built. Now we need to define how our users will be added. For this script we are going to run 10 concurrent users for 30 seconds: setUp(myScenario.inject(constantUsersPerSec(10) during (30 seconds)).protocols(httpConfig)) We'll also need to import the DurationInt from Scala so that seconds will resolve:\n1import scala.concurrent.duration.DurationInt Gatling provides a number of ways to add users. Checkout out their documentation for more details.\nAnd there we have it. The final product should look like this: Running the script We have a couple of ways to run the script. The easiest way is likely from the command line with Maven:\n1mvn gatling:test -Dgatling.simulationClass=scala.scripts.HelloGatling Another option is to run src/test/resources/Engine.scala. This is the same utility we used in the first post when we ran Gatling from the bin folder.\nRegardless of the method you choose, your newly created script should run and the results should look something like this: Wrap-up So there is your first script from scratch. It's not a whole lot different from the recorded script but hopefully you have a better understanding of how to create a new Gatling project with Maven, and some clarity on the inner workings of a script.\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/performance-testing/automate-your-load-testing-with-gatling-the-sequel/","section":"blog","tags":["load testing","performance testing"],"title":"Automate Your Load Testing With Gatling - The Sequel!"},{"body":"I first got involved with automated load testing about 10 years ago. At that time I was working as a developer on a team building an enterprise application to be used by hundreds of concurrent users.\nTo prepare for load testing we first setup a production-like test environment. We then happily navigated to the home page to do some basic smoke testing before the real testing began. Unfortunately the home page would not load at all.\nAfter some investigation, we realized we had SQL performance issues with just a single user, and our application needed to support hundreds of concurrent users.\nThis was my introduction to automated load testing and tuning, and helped me understand why automated load testing is so critical to delivering quality enterprise software.\nWhat is Load Testing Load testing validates performance-related non-functional requirements. Sample performance requirements could include:\nThe application must support up to 300 concurrent users Response time must be less than 3 seconds 95% of the time Response time must be less than 5 seconds 99% of the time Other inputs to load testing could be the business transaction distribution. For example:\nSearching inventory is 45% of all transactions Placing a new order is 35% of all transactions Checking order status is 10% of all transactions Administrative tasks are 10% of all transactions Based on these types of requirements, load testing tools enable us to design and develop automated processes to verify the application conforms to the requirements, and to identify areas that need attention.\nTypically the purpose of the testing is to ensure the backend environment is designed, developed, and sized in the way that will allow the application to meet performance requirements.\nTo execute a 300 concurrent user load test, we don’t need to open 300 versions of a browser or mobile app. Instead, our load testing tool will simulate the client by sending the same API requests the client would send. What is Gatling Gatling is a popular load test tool with two versions, an open-source version that is free, and an Enterprise version, Gatling FrontLine. For this post I will be focusing on the free version.\nGatling is written in Scala so it can run on both Linux and Windows machines. Gatling is a code-based tool, which means you are writing Scala code to build your scripts. Gatling does not have an IDE; rather it’s a plug-in that will integrate with popular IDEs such as Visual Studio Code or IntelliJ IDEA.\nGatling is built utilizing the Akka toolkit, meaning Gatling does not need a separate thread for each user, allowing for a much larger workload to be generated on a single test machine as compared to tools like JMeter.\nGatling provides robust reports that allow you to analyze how your application is performing under load.\nOK, enough talk, let’s start using Gatling!\nRunning a sample Gatling script Download Gatling Access the Open Source page and click the Download Now button. As of this date, the current Gatling version is 3.3.1, so this will download the sample project: gatling-charts-highcharts-bundle-3.3.1-bundle.zip. Extract the downloaded zip file. Run the sample script Gatling includes some sample scripts ready for you to execute. From a command line, navigate to the /bin folder of your unzipped working folder and execute ./gatling.sh (gatling.bat on Windows) Gatling will then ask you a couple questions before running the script. For now select simulation “0”, enter a run description if you like, and press enter. The script will start and run for about 25 seconds. If things go as planned you should see something like this: In addition to the stats in the command window, you’ll see a location for the HTML results report that you can open in a browser. It should look something like this: So that’s it! You downloaded a sample gatling script, ran it, and viewed the results. If you want to take a look at the script you ran, you can find it at user-files/simulations/computerdatabase/BasicSimulation.scala. Open it with a text editor or your favorite IDE.\nNow it’s time to create your own script!\nYour First Script For our first script will test a sample Gatling website. This site provides a list of computers in inventory, and allows you to add and change computers. We want a script that will view the list of all computers and then add a new one.\nThere are a couple options for creating Gatling scripts. One is to use the Gatling Recorder that is included with Gatling, and the other is to develop scripts from scratch. Since we're just getting started, let's start with the recorder.\nUsing the Gatling Recorder To start the Gatling Recorder, from a command line navigate to the /bin folder of your unzipped working folder and execute ./recorder.sh (recorder.bat on Windows). The recorder utility will launch: We're going to use this utility to generate a test script.\nIn the upper right corner there is a dropdown for Recorder Mode that has 2 options, HTTP Proxy and HAR Converter. For this post we are going to select HAR Converter, which uses a HTTP Archive File as input the the script generation process.\nGenerating a HAR File A HAR file contains detail about the browser's interaction with network resources. The file is typically used to help troubleshoot performance issues, as it contains detailed performance data from the browser perspective. The HAR file is in JSON format and can be generated with Chrome, Firefox, IE, or Edge. We are going to generate a HAR file using Google Chrome.\nOpen Chrome, open Developer Tools, click the Network tab, and browse to https://computer-database.gatling.io/computers. We are going to generate a load test script for this site.\nIn Developer Tools, click the Clear button, then click the Preserve Log checkbox, which will allow your network activity to be captured as a HAR file.\nNow we are going to step through our site and allow the browser to record the activity to the HAR file.\nRefresh the page Click the green \u0026quot;Add a new computer\u0026quot; button Fill out details for the new computer and click \u0026quot;Create this computer\u0026quot; Now we just need to save the HAR file. Right click anywhere in the network activity window and select \u0026quot;Save all HAR with content\u0026quot;. Now that you have a HAR file, you can use that in the recorder.\nGenerating the script with the HAR file Back in the recorder, let's generate the script from the HAR file:\nFor Recording Mode, select \u0026quot;HAR converter\u0026quot; For \u0026quot;Har File\u0026quot;, select the HAR File that you saved previously For \u0026quot;Class Name\u0026quot;, enter MyFirstGatlingScript Leave all other options as is, and click the Start button at the bottom right. Hopefully you see something like this: View the script From the Gatling install folder, navigate to the user-files/simulations folder. You should see a file named \u0026quot;MyFirstGatlingScript.scala\u0026quot;. It's a text file, so open it in your favorite text editor.\nI'm not going to cover everything in the script, but there are a few points I'd like to highlight:\nAbout halfway into the script you should see some code like this: Even if you're not familiar with Scala, this may seem fairly familiar. Here you can see the server calls that were made when you were stepping through the site to generate the HAR file:\nrequest_0 is .get(\u0026quot;/computers\u0026quot;), which was the display of list of computers. request_1 is .get(\u0026quot;/computers/new\u0026quot;) which was the click of the \u0026quot;Add a new computer\u0026quot; button. request_2 is .post(\u0026quot;/computers\u0026quot;) which is adding the new computer. After request_0 and request_1 you should see a pause statement, which will introduce a pause in the script to simulate user think time. The number of seconds to pause is based on how long you paused between steps when the HAR file was initially created. This data was automatically saved when the HAR file was created by Chrome. Realistic think time in critical to building realistic load test scenarios.\nThe final point of interest is the last line: This code is setting up how many concurrent users to run and how to add them to the test. For this example, we have just a single user that will be added immediately.\nRun the script Finally, let's run this new script. We'll follow the same patten we used to run the initial sample script:\nFrom the Gatling install folder, navigate to the bin folder\nExecute ./gatling.sh (gatling.bat on Windows)\nYou should now see a new script in the list of available scripts, and that's the new script you just created: Select your script and watch it run!\nOnce the script ends you should see something like this: And there you have it. You create your first Gatling script and executed it!\nThis is a basic introduction to Gatling. In the real world you won't be able to rely on the recorder to do all the work.\nYou'll need a strategy for introducing test data, asserting API calls work as expected, including multiple business scenarios, logging, etc.\nYou are going to have requirements for your tests that will require you to edit the generated scripts, and you will likely need to build new scripts from scratch. In my next post I'll get into details like this and more, so stay tuned!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/performance-testing/automate-your-load-testing-with-gatling/","section":"blog","tags":["load testing","performance testing"],"title":"Automate Your Load Testing With Gatling"},{"body":"I recently worked with an organization that was looking to make changes to their software delivery processes so they could deliver higher quality software more quickly. We used research and guidance from Google’s 2024 DORA Report to drive those changes.\nThe goal of the report is to identify ways that technology can deliver value to organizations and their customers. The content of the report was driven by yearly surveys completed by over 31,000 IT professionals worldwide.\nOne area of the report I found especially interesting was the survey questions related to the Four Key Metrics of software development.\n##Throughput survey questions ###Deployment frequency For the primary application or service you work on, how often does your organization deploy code to production or release it to end users? ###Lead time for change For the primary application or service you work on, what is your lead time for changes (i.e., how long does it take to go from code committed to code successfully running in production)?\n##Stability survey questions ###Time to restore service For the primary application or service you work on, how long does it generally take to restore service when a service incident or a defect that impacts users occurs (e.g., unplanned outage or service impairment)? ###Change failure rate For the primary application or service you work on, what percentage of changes to production or released to users result in degraded service (e.g., lead to service impairment or service outage) and subsequently require remediation (e.g., require a hotfix, rollback, fix forward, patch)?\nThe table below captures the results of the survey.\nAspect of Software Delivery Performance Elite High Medium Low Deployment frequency On-demand (multiple deploys per day) Between once per day and once per week Between once per week and once per month Between once per month and once every six months Lead time for change Less than one day Between one day and one week Between one week and one month Between one month and six months Time to restore service Less than one hour Less than one day Less than one day Between one week and one month Change failure rate 0-15% 0-15% 0-15% 46-60% *This table is provided courtesy of the ACCELERATE state of DevOps 2019 report Looking at the results, a couple things stand out to me:\nElite performers deploy frequently. Elite performers release bugs to production just as frequently as High and Medium performers. The difference is they get them fixed a lot more quickly. Comparing Elite performers to Low performers, the 2019 report finds:\nElite performers have 20 times more frequent code deployments Elite performers lead time from commit to deploy is 106 times faster Elite performers recover from service incidents 2,604 times faster Elite performers are 1/7 as likely to deploy failing changes ##Wrap-up These metrics are a great way to get a baseline of your organization's software delivery performance, and a good way to quantify the success of your agile transformation activities.\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/measuring-your-teams-software-delivery-performance/","section":"blog","tags":["devops"],"title":"Measuring Your Team's Software Delivery Performance"},{"body":"If you've worked in QA automation for awhile you have no doubt heard of the test pyramid. The test pyramid is a diagram used to visually depict test types, and to give some general guidelines on how many of each to create.\nAs you can see from the pyramid, a well-rounded automated test suite should have a large number of unit tests, fewer API or integration tests, and even fewer end-to-end UI tests. Unit testing is the handled by the developer, UI testing is handled by QA, and API testing is typically a somewhat shared responsibility.\nAt times a QA team will focus on UI testing and ignore API testing. It's easy to understand the infatuation with UI tests:\nUI tests are easy to map to user stories. The business user can easily relate to UI tests. Most folks working in QA automation have more experience with UI automation. UI tests are a lot more fun to demo compared to a command line API test. So why do we want more API tests that UI tests? I can think of at a number of reasons:\nAPI tests allow for more code coverage, since it's easier to test different code paths and error situations. API tests run much faster, so we can get faster feedback to the developer. API tests can find bugs earlier, before the UI layer is created. API tests can be created before the API is developed by using mocking frameworks. API tests are less brittle, therefore less costly to maintain. API tests are necessary for APIs that don't have a UI, such as APIs that are consumed by IoT devices or other services. There are a number of tools that can be used to test APIs, such as SoapUI and REST Assured. Another is Postman. I have used Postman for a number of years for quick manual testing of REST services, but only recently have I really started looking at everything it provides. So what is Postman?\nPostman is an HTTP client marketed to both developers and testers to support development and testing of APIs. Postman is available as a standalone client, and also as a Chrome app. Google has announced plans to end support of Chrome apps in the near future, so the native standalone app is the way to go.\nCollections Everything with Postman starts with collections, which is how you store and categorize your individual API requests. Collections allow you to organize your API requests in folders and subfolders, and the requests can be run together in Postman via Collection Runner.\nPre and post scripts Postman allows automation developers to use JavaScript in any API request to interact with the API request, API response, and global and environmental variables. These scripts are typically used to pass data between related API calls in a collection, and to verify the results from the API request match the expected results. For example:\nVerify the response status code: 1pm.test(\u0026#34;Status code is 200\u0026#34;, function () { 2 pm.expect.response.to.have.status(200); 3}); Verify the response time of the request: 1pm.test(\u0026#34;Response time is less than 500ms\u0026#34;, function () { 2 pm.expect(pm.response.responseTime).to.be.below(500); 3}); Verify the data returned by the request: 1pm.test(\u0026#34;Customer Name Updated correctly\u0026#34;, function () { 2 pm.expect(jsonData.lastName).to.eql(\u0026#34;Smith\u0026#34;); 3}); Environmental variables As you build and run tests, you'll typically need to run the tests in a number of environments, such as local, dev, test, UAT, etc. Postman provides an easy mechanism to define specifics about each of your environments and to quickly switch between them.\nTest and develop in parallel One of the drawbacks with UI testing is that you need the developer to develop the functionality before you can really start building the the guts of the test automation. With API testing, you can start testing when the API is complete, even if the front-end components have not been started.\nAdditionally, with mocking you don't even need the API to be complete. Postman provides a mocking service that allows you to define an API endpoint with the expected input and output. When the sprint starts you can build and run your automation tests against that mock service. Once the development team completes the API, you can just point to the API and run your test.\nProxy server to capture API calls If you're looking to quickly identify and build a test process for an existing business transaction, Postman provides a mechanism to capture the API traffic for a particular transaction and store the API calls in a Postman collection. Just turn on Postman's transaction interceptor, manually step through the transaction via the UI, and the API calls will be captured in a Postman collection.\nAutomating API test with Newman CLI Newman is a command line tool, written on Node.js and stored in the NPM repository. Once it's installed, you have the ability to run a Postman collection from the command line. With Newman you can incorporate API testing and custom reporting into the CI/CD pipeline.\nTeam Collaboration One of the key features of Postman Pro is sharing and team collaboration. Postman collections can be shared and edited by authorized team members via a team API library in the cloud. The team activity feed allows you to see all modifications made to a collection and to rollback to a previous version when necessary.\nPricing Postman comes in 3 pricing plans: Free, Pro, and Enterprise. The core features for testing are included in the Free version. The Pro version offers options for team collaboration in the cloud, while the Enterprise version includes extended support and higher cloud usage limits. If you're just getting started, the Free version will give you everything you need, and there are trial versions of Pro and Enterprise when you are ready.\nNext Steps In this post I have only scratched the surface regarding some of the cool features that Postman provides To learn more about Postman and API testing, the Postman website is a great place to get started. Also, Postman has a yearly tech conference and the sessions and talks from last year are available to view for free.\nFinally, don't get so enthralled with UI testing that you lose sight of API testing. Be like the test pyramid; spend a lot more time with API testing than you do with UI testing!\n","link":"https://dennis-whalen-dot-com.vercel.app/blog/postman/api-testing-with-postman/","section":"blog","tags":["postman","api"],"title":"API Testing With Postman"},{"body":"","link":"https://dennis-whalen-dot-com.vercel.app/archives/","section":"","tags":null,"title":""},{"body":"Apps I'm designing and building. Each one starts with a simple question: what would make everyday life a little easier?\n","link":"https://dennis-whalen-dot-com.vercel.app/apps/","section":"apps","tags":null,"title":"Apps"},{"body":"Simply Today is a calm daily planner for weekends, retirement days, and any day that only needs a little shape. Simply Today is not another calendar.\nWhether you're adjusting to life after work or just want Saturday and Sunday to feel intentional, you build a personal list of activities and each day choose a few for Morning, Afternoon, and Evening. That's the whole system.\nWhy Simply Today? Only today and tomorrow are on the table—no far-future scheduling or backlog guilt No meeting blocks, no minute-by-minute planning No reminders or notifications nagging you No streaks, scores, or habit tracking—just planning How it works Add the activities that make your days feel good Each day, place a few into Morning, Afternoon, or Evening Done—open the app when you want to adjust your plan Who it's for Whether you're adjusting to retirement, shaping weekends, or just tired of days that drift, Simply Today offers a light framework: What do I want to do today?\nYour data Sign in with Apple, Google, or email. Your plans sync to your account. You can delete your account and data anytime from the app.\nSimply Today is intentionally small: planning for real life, without the noise.\n","link":"https://dennis-whalen-dot-com.vercel.app/apps/simply-today/","section":"apps","tags":null,"title":"Simply Today"}]