Mistral AI API: Unleashing the Power of Mixtral 8x7B and Mistral Medium
Today, I got access to the Mistral AI API, and I want to do some testing comparing it to 3.5 and also GPT 4. I also have my usual prompts where I try to test some reasoning skills, maybe some math and coding problems. So, I think we just need to get into it and see what Mistral AI can deliver. Let's take a quick look at the platform.
The Platform
The Mistral AI platform has a simple setup with options for home building, API keys, documentation, and API. The documentation provides a client code and installation instructions for Python. It also offers chat completion options, including streaming and non-streaming. The safe mode allows for secure setups, and there is an embeddings model as well. For today's testing, we will focus on the three available models: the tiny model, the small model powered by Mixtral 8 * 7B, and the medium model powered by an internal prototype model.
Before we dive into testing, let's take a look at the pricing. The medium model is priced at €7.5 (approximately $8.25) per 1,000 tokens. The small model is priced similarly, making it competitive with GPT 3.5 Turbo. Now, let's proceed with our tests.
Test 1: The Shirt Problem
The first test we will run is the shirt problem. This problem involves hanging shirts out to dry in the sun and determining how long it will take to dry a specific number of shirts. We have previously tested various models on this problem, and not all models have been able to solve it correctly. Let's see how Mistral AI performs.
First, let's test GPT 3.5:
- GPT 3.5: It incorrectly concludes that it will take 20 hours to dry 10 shirts, not considering that drying time is not dependent on the number of shirts.
Now, let's test Mistral AI on the small model:
- Mistral Small: It correctly concludes that if the conditions for drying, such as sunlight, are the same, it will still take 10 hours to dry 10 shirts. Multiple shirts do not increase the drying time as long as they do not overlap or block each other.
Next, let's test Mistral AI on the medium model:
- Mistral Medium: It also correctly concludes that it will take 10 hours for 10 shirts to dry, considering the same conditions as the small model. Mistral AI performs well on this problem.
Overall, Mistral AI has shown better reasoning skills compared to GPT 3.5, providing accurate answers to the shirt problem.
Test 2: The World Problem
The second test we will run is the world problem. This problem involves tracing the path of an object, such as a ball, through a series of steps and determining its final location. Let's see how Mistral AI and other models perform on this problem.
First, let's test GPT 3.5:
- GPT 3.5: It incorrectly concludes that the ball is ultimately in the sealed box on its way to your friend in London, not considering the fact that the ball fell out through the hole in the bag earlier.
Next, let's test Mistral AI on the small model:
- Mistral Small: It does not provide specific information about the ball's location but suggests that it could be lost during shipping or still in the bag. This answer is not what we are looking for.
Now, let's switch to the Mistral AI medium model:
- Mistral Medium: It provides a more accurate answer, suggesting that the ball is most likely not in the box that arrived at your friend's address in London. Considering that the bag has a hole, it is likely that the ball fell out during transit. This aligns with our expectations and shows reasonable reasoning.
GPT 4, known for its strong reasoning abilities, also provides the correct answer, suggesting that the ball is on the floor in your office in New York. Overall, Mistral AI, particularly the medium model, demonstrates effective reasoning skills on this world problem.
Test 3: Python Coding Problem
The third test we will conduct involves a Python coding problem to create a snake game with a user interface for Windows. Let's see how Mistral AI and other models perform on this problem.
Let's start with GPT 3.5:
- GPT 3.5: It does not provide the full code as requested. While the generated code looks okay for step-by-step execution, it falls short of the complete code.
Next, let's test Mistral AI on the small model:
- Mistral Small: It generates the code step-by-step, but it does not provide the full code as requested.
Now, let's switch to the Mistral AI medium model:
- Mistral Medium: It generates the code using the Tinter library, providing a better UI for the snake game with a score display. However, the ball seems to disappear, affecting the game's functionality.
GPT 4 performs the best in this test, providing the complete code for the snake game with an improved UI and proper functionality.
Streaming Function
It is worth mentioning the streaming function available in Mistral AI. This feature allows for streaming requests, similar to OpenAI. Let's compare the streaming capabilities of the Mistral AI models.
First, let's test the streaming function with the tiny model:
- Tiny Model: The streaming function provides quick and responsive results.
Next, let's test the streaming function with the small model:
- Small Model: The streaming function is slightly slower than the tiny model but still provides relatively quick results.
Finally, let's test the streaming function with the medium model:
- Medium Model: The streaming function is slower than the previous models but still offers reasonably fast results.
The streaming function in Mistral AI is a valuable feature that enhances the user experience and allows for efficient interaction with the API.
Conclusion
In conclusion, Mistral AI proves to be a promising API for various tasks, including reasoning, problem-solving, and coding. The small and medium models demonstrate superior reasoning skills compared to GPT 3.5, providing accurate answers to complex problems. The pricing is competitive, making Mistral AI an attractive option for developers. The streaming function is a convenient feature that enhances API usage. Overall, I am positive about the capabilities of Mistral AI and excited to explore more APIs in the future.
Frequently Asked Questions
-
1. Can Mistral AI handle math problems?
Yes, Mistral AI has demonstrated the ability to solve math problems effectively, providing reasoning and accurate answers.
-
2. Is Mistral AI more reliable than GPT 3.5?
Based on our testing, Mistral AI, particularly the small and medium models, have shown better reasoning skills and accuracy compared to GPT 3.5.
-
3. Are there any known limitations of Mistral AI?
While Mistral AI performs well on a range of tasks, it is important to note that its performance may vary depending on the complexity of the problem and the specific model used.
-
4. Can I use Mistral AI for natural language processing tasks?
Yes, Mistral AI is capable of handling natural language processing tasks, providing accurate and contextually relevant responses.
-
5. Is Mistral AI compatible with other programming languages?
Mistral AI primarily supports Python, but there may be options to use it with other programming languages through appropriate integrations or wrappers.
