Open Source Friday with API Bench - Performance-test anything!

GitHub Models

• The video discusses GitHub Models, a platform for exploring and experimenting with AI models.
• Users can interact with models like GPT-40 and 53 mini instruct, adjusting parameters and sending prompts to simulate real-world applications.
• GitHub Models provides access to model details, including documentation, evaluations, and transparency information.
• Users can integrate models into their code using GitHub code spaces, which provide preconfigured development environments with SDKs and samples.
• GitHub Models leverages entitlements linked to GitHub accounts, eliminating the need for API keys or additional sign-ups.

Bjorn Butner and API Bench

• The video then transitions to introduce Bjorn Butner, a software engineer who created API Bench, an open-source project for API testing.
• Bjorn has been a software engineer for 17 years and got involved in open source through bug reporting for a computer game.
• He started developing API Bench four years ago because he needed a tool for API testing.
• Bjorn works on API Bench in his free time, using it for testing at his full-time job as a web developer.

API Bench: Purpose and Features

• API Bench was created to address the issue of performance testing in a large codebase with millions of lines of code.
• The developer faced challenges with existing open-source tools, including lack of documentation and maintenance.
• API Bench is a rewrite of these libraries, built with TypeScript for better user support and features like code generation and plug-and-play functionality.
• API Bench provides a set of standard tests and processing tools, including user agent filtering.
• The tool generates a Benchmark folder containing the necessary files for testing, including a pre-generated TypeScript file.
• Users can create new JavaScript or TypeScript files within the Source folder to define their tests.
• API Bench automatically injects environment variables into the test functions.
• API Bench is compatible with any API built with any language, as long as it is REST-like, including JSON APIs and APIs using form URL encoding.
• It supports a couple of databases for storing run data for later review and comparison.
• API Bench requires minimal JavaScript knowledge, with the basic user request being a JSON object.
• It can handle large volumes of API requests, but the current implementation has a technical limitation due to the creation of an array of requests.
• Stress testing is planned for a later release this year.
• API Bench is a one-stop solution for performance testing APIs, providing valuable insights and tools for optimizing API performance.

API Bench: Functionality and Benefits

• The demo showcases API Bench's ability to test fake APIs, providing statistics like average response time, standard deviation, and minimum/maximum values.
• The standard deviation helps determine the stability of the API, indicating whether there are significant variations in response times.
• API Bench offers a CLI tool for generating the base structure and test code for different types of requests (GET, POST, PATCH).
• The CLI tool allows for reusability by extracting the root URL as an environment variable.
• The framework includes pre and post-processing for tasks like JWT authentication and status code checks.
• API Bench supports JSON data format and is planning to add XML support in the future.
• The video demonstrates how to use API Bench, an open-source tool for performance testing.
• The video highlights the importance of limiting the number of requests to avoid overloading the server.
• The video demonstrates how to create a simple test function and run it using npm start.
• The output of the test includes statistics such as average, median, maximum, and minimum response times.
• The video explains that these statistics provide insights into the performance of the API under different conditions.
• The video emphasizes the importance of understanding the standard deviation, which indicates the variability of response times.
• API Bench is an open-source project that allows users to performance test their APIs.
• It provides information about API performance, including statistics, load testing capabilities, and integration tests.
• API Bench offers features like standard deviation measurement, load testing, integration tests, and content tests.
• It allows users to blacklist middleware for specific test types, reducing the effort required for different testing scenarios.
• The main advantage of API Bench is the small amount of code required to achieve good test coverage.
• It allows for different tests to be executed from the same test code without any changes.
• API Bench is easy to integrate into CI pipelines and is built for a command-line interface, making it easier to use than other tools.
• Users can import specific requests from tools like curl, OpenAPI documentation, and H files.
• API Bench is not intended for requests that differ significantly, but users can create custom requests with random values.
• It is primarily intended for performance testing, where consistency is important for accurate comparisons.
• API Bench can be used with various programming languages, including Python, PHP, and Java.
• It works by measuring the output of the API without attaching to the process, similar to tools like Insomnia or Bruno.
• API Bench is designed for automation and can be easily customized per repository or project.

API Bench: Integration and Future Development

• API Bench can be integrated into CI/CD pipelines using Docker containers for GitLab, GitHub, and G.
• It also has a Jira integration that allows users to post test results to Jira.
• Upcoming features for API Bench include the ability to merge results from multiple machines and expand integration with other platforms like GitHub.
• API Bench is working on adding more monitoring features, including a self-hosted history website for viewing API performance graphs and detailed run information.
• The team is also improving the core functionality of API Bench by refactoring the old core and replacing an array with a generator to reduce memory usage.
• Future improvements include a desktop client built with Electron, which will require less coding and command-line interaction.
• API Bench currently supports four languages, and contributions in the form of translations are welcome.
• The tool allows for easy authentication token handling through middleware, which automatically injects the token into the headers.
• API Bench supports various JSON structures for access tokens and allows users to add their own structures through pull requests.
• The developer experience was a priority during the development of API Bench, which is why it uses TypeScript for autocompletion and type safety.
• The tool provides a Task type for tests, allowing for type checking and improved code completion.

API Bench: Open Source and Community

• API Bench is a performance testing tool that allows users to test the performance of any API.
• It supports various databases, including MySQL, MariaDB, PostgreSQL, and MSSQL.
• API Bench is open source and welcomes contributions from anyone interested in learning and sharing knowledge.
• The project has a core team of 10 people and around 50 total contributors, including non-code contributors.
• The documentation website automatically pulls code contributor information from the GitHub API and includes a JSON file for non-code contributors.
• The speaker encourages viewers to explore the documentation and implement API Bench in their own APIs to improve performance and security.