In our post “AI in SAP Consulting: Are SAP Consultants Now Obsolete?” we tested four AI solutions with 17 expert questions on SAP Public Sector Management (PSM). The result for SAP Joule for Consultants (J4C) was sobering: SAP’s own AI answered only 8 of 17 questions correctly. The best solutions we tested reached 12 correct answers. Even a general-purpose AI model with no SAP specialization achieved 8 correct answers.
Since J4C is marketed as being built on exclusive SAP knowledge, we wanted to dig deeper. We ran the test again, this time with the new expert feature, more detailed answers, and additional domain knowledge from our PSM consulting practice.
To say it up front: the expert role alone does not improve the technical quality. What matters is the knowledge you provide to the AI.
What SAP Joule for Consultants Can Do and Which Features We Tested
With Joule for Consultants, SAP offers its own AI solution focused on SAP-specific knowledge. A web or resource search runs automatically, but only official SAP sources are included. SAP also provides a prompt library in which favorites can be marked. However, you cannot save your own prompts there.
Another feature is the expert area: here you can adopt existing SAP experts or create your own, with the help of a starter template. Examples include experts for utilities, logistics, or clean core. This is most comparable to roles or custom system prompts in other AI tools. In addition, extra documents can be attached to an expert: up to ten files per expert, with 1 GB of total storage across all experts.
The experts are by no means comparable to skills as known from Claude, for example: anyone hoping to reuse existing .skill files will currently be disappointed. Markdown files, on the other hand, can be uploaded.
What’s New: from Chat to a Custom Expert
SAP develops J4C further in short release cycles. The following new features were most relevant for our test:
- Expert Workspace: Custom experts can be configured with a name, description, up to three tags, and custom instructions of up to 10,000 characters. Attached files are permanently available in all conversations with the expert.
- Expert Library: Expert templates prepared by SAP, for example for Clean Core, Finance, or Logistics, can be adopted and customized.
- Starter-Prompts: Up to four predefined opening questions can be stored per expert to start conversations faster.
- AI-assisted revision: With “Optimize,” Joule rewrites the instructions. “Compare with entered text” lets you see what changed.
- Sharing and exporting: Experts can be shared with colleagues in the same J4C tenant and exported.
- Custom Knowledge Grounding: At tenant level, J4C can additionally be connected to company-owned content, for example from SharePoint, Amazon S3, or SAP Build Work Zone. We did not use this option in our test.
- Convenience features: These include a Prompt Guide with best practices, Quick Links, and the ability to edit prompts after they have been sent.
Our Test Setup: an SAP PSM Expert for the German Market
Fittingly, we created the expert using J4C itself. As a basis we used the structure SAP provides for custom instructions, covering role and scope as well as approach and structure, among other things. We added our own requirements: answers in German, at executive-consultant level, and with a clear focus on the German public sector, including the budgetary principles under BHO/LHO and the SAGA standards.
The resulting instructions take up about 5,300 of the possible 10,000 characters. We also added tags for budget, funds, and grants management, as well as starter prompts, for example on the migration to the Budget Control System (BCS).
We then asked J4C the same 17 questions as in our first test in three stages. This time the rule was to answer each question in up to five lines. In our first test, answers were limited to two lines per question.
- Without an expert
- With the SAP PSM expert activated.
- With the SAP PSM expert and additional attachments: nine Markdown files totaling about 207 KB. We developed these with Claude based on our adesso SAP PSM knowledge, specifically from our Claude skill “SAP PSM Consultant.” Its SKILL.md and reference files cover budgeting, master data, customizing, grants, integration, BAdIs, and reporting, among other topics.
The Result: the Expert Role Alone Isn’t Enough – Knowledge Makes the Difference
As in the first test, the answers were evaluated against the same model solutions. The following overview shows the number of correct answers per variant:
| Variant | Correct answers (out of 17) |
| SAP J4C (first test), up to two lines per question | 8 |
| SAP J4C, up to five lines per question | 8 |
| SAP J4C with SAP PSM expert, up to five lines per question | 8 |
| SAP J4C with SAP PSM expert and adesso knowledge as attachments | 12 |
Neither the longer answers nor the expert alone improved the result. The answers were more structured and included summaries, analyses of the PSM components, and notes on risks and configuration paths. In substance, however, the errors remained. The professional presentation sometimes made wrong answers seem even more convincing.
Only with the attachments did J4C improve significantly: with 12 of 17 correct answers, it reached the level of the best solutions from the first test. Four questions that had previously been answered incorrectly were now correct: on master data in grants management (grantee), on whether classic budgeting and Grantee Management can be combined, on the BCS customizing steps, and on budget subtypes in BCS.
The effect is particularly clear in the question of whether classic budgeting can be used with Grantee Management. Without attachments, J4C considered the two budgeting approaches generally combinable. With attachments, the answer was correct: “No – Grantee Management strictly requires BCS, and classic budgeting has been deprecated under S/4HANA anyway.”
The budget subtypes followed a similar pattern: previously, J4C confirmed that budget subtypes exist in BCS and backed this up with invented examples. With attachments, J4C correctly explained that the budget subtype of classic budgeting is replaced in BCS by customer-defined budget types.
Some questions were not answered correctly in any variant, for instance on updating postings between two commitment items with financial transaction 30, on PAYAC, or on prohibited business transactions. This shows that even well-prepared foundational knowledge does not fully replace deep project and detail knowledge.
Limitations in Practical Testing
In the process of building a better expert, we ran into several limitations:
- Limited number of attachments: A maximum of ten files per expert and 1 GB of storage across all experts. Anyone bringing an extensive skill with many reference files has to consolidate the content.
- No direct use of skills: Skills as known from other AI tools cannot be used directly. Their content has to be uploaded as Markdown files. The control logic of a skill, that is, when to draw on which reference, is lost in the process.
- Processing times and timeouts: While changes are being processed, an expert is read-only. In longer sessions we also experienced time overruns and session timeouts.
In theory, the expert you created can be tested right away. In our test, however, very often nothing happened: Joule showed “Thinking,” but even after many minutes it was not ready for input. This frequently ended in a timeout.
Conclusion: In Practice, Curated Knowledge Makes the Difference
Our test shows that the expert feature in J4C is a useful addition, but not a self-runner. A role or a system prompt alone does not automatically improve technical quality in our test. What makes the difference is curated expert knowledge: with the adesso PSM content as an attachment, J4C reached the level of the best solutions from our previous SAP PSM test. The available SAP knowledge alone was therefore not sufficient for our question set. It remains an open question whether the additional effort of preparing the knowledge is worthwhile given J4C’s comparatively high license costs.
For customers and consulting firms, the takeaway is clear: value comes from structuring and providing verified project knowledge to the AI – whether as an attachment in J4C, as a skill or in a custom AI solution. This knowledge remains the core of good SAP consulting and forms the foundation for adesso business consulting to apply AI in SAP projects with domain expertise and real added value.
