DeepSee Troubleshooting Guide

The goal of this “DeepSee Troubleshooting Guide” is to help you track down and fix problems in your DeepSee project.

If the problem can’t be fixed by following the guidelines, you will at least have enough information to submit a WRC issue with DeepSee Support and provide all the evidence to us, so we can continue the investigation together and resolve it faster!

Please NOTE: If you are unfamiliar with the consequences of a given action or command, please don’t run it as it might have an effect on your production system.  In this case contact DeepSee Support for further assistance. 

The easiest way to follow this guide is by starting on the left and make your way to the Action column on the right.

Problem AreaWhat happened?AnalysisSolutions
Build ProblemsErrors during buildsCheck Error Message
Run $System.OBJ.DisplayError() if Build didn't show errors
Check ^DeepSee.BuildErrors/ run ##class(%DeepSee.Utils).%PrintBuildErrors(pCube)
Fix Errors in Message
Less records built than rows in source tableCheck cube for build restrictionremove or accept build restriction
Check if maxfacts is usedremove maxfacts
Check ^DeepSee.BuildErrors/ run ##class(%DeepSee.Utils).%PrintBuildErrors(pCube)fix build errors
Check indices in source classRebuild indices in source class
Missing DataRecords from source are not available in DeepSeeCheck previous section for problems / restrictions in buildremove restrictions
Check "%OnProcessFact" in cube definitioncorrect method if necessary
Check for build errorsfix build errors
Debug methods in sourceExpressionfix methods
Check Fact table recordsfind cause of mismatch and resolve it
Compare Fact table to source table 
Wrong ResultsQuery shows unexpected resultsCheck previous section if data is missing
Check if caching problem:
 - do $System.DeepSee.Shell() 
 - cache off
Fix errors on build
do $System.DeepSee.Reset()
kill ^DeepSee.Cache
Split query in smaller segments (i.e. each axis separately)
Find problematic segment
fix Problem in Segment
Check ^DeepSee.AgentLogdo ##class(%DeepSee.TaskMaster).%Reset()
Listings are emptyCheck if normal ListingGrant permissions on selecting from source table
Check if Custom SQL ListingDebug WHERE conditions
Check ^DeepSee.SQLError
Check ^DeepSee.QueryLog
Hangs / Unexpected EventsDeepSee not reacting as expectedCheck System Health 
hangsCheck agent status 
Check License UsageBuy More Licenses
previously working operations failCheck ^DeepSee.AgentLogdo ##class(%DeepSee.TaskMaster).%Reset()
Run do ##class(%DeepSee.TaskMaster).%PrintLog() 
Check ^DeepSee.LastLogError
Check ^DeepSee.PivotError()
Fix Errors found
Performance IssuesBuild Performance lowRun do ##class(%DeepSee.TaskMaster).%PrintLog() 
Check number of available agentsdo ##class(%DeepSee.TaskMaster).%Reset()
Check memory and CPU usagerun build in phase of low load
Check source expressionsavoid source expressions if possible
Check general Cube Statistics:remove unnecessary dimension/levels
 - do ##class(%DeepSee.Utils).%Analyze("Holefoods")keep cubes simple and slim
High journaling activityUsing a data warehouse design or separate namespace for DeepSee you may disable journaling in this namespace
Query Performance lowrun Report: do ##class(%DeepSee.Diagnostic.MDXUtils).%Run(<query>)plausible long running queries need simplification or workaround
Check memory and CPU usagefree resources
Check buffer settingsraise buffers if possible
Make general System Perfomance Analysison runtime perforrmance issues ask WRC for help
Common Reported IssuesNamespace Not Listed In DS MenuCheck if DeepSee is enabled  in web-application settings /csp/<namespace>enable "DeepSee" in web-application
Architect is grayed-outCheck licenseget DeepSee enabled license
Check roles of useradd resource %DeepSee_Architect / %DeepSee_ArchitectEdit Use
Check if your browser is not IE 8use supported browser

Here are the mind-map diagrams for each problem area:

  • + 11
  • 0
  • 1126
  • 16

Comments

Great work, Sascha! Thank you!

Is it possible to make a mindmap version of this table too?

I need to check which other visualizations  will work. I think the table gives the best overview. A mindmap or flowchart could look confusing.

As I see there are some cases with empty boxes in the solution column. I would add "Put a question on the Developer Community" there as a solution ;)

 

Fixed a bit style for table, increased font-size, and removed unnecessary empty cells

Thanks Sascha and Dmitry!  

It would be nice to add links (where applicable), to either documentation or other posts in the Developer Community, that have more details about the topic/troubleshooting step.

 

Sascha!

That looks just great! It's significantly clearer now. I'll print it out, it would be my favorite wall poster ;)

 

I would add also to the Build/Less facts section following:

sometimes you have no facts for the cube against source class with plenty of records. Please check the indexes in the source class and rebuild it, it can help. I met this thing.

I added your suggestion in the table and mind map.

Great guide. I would add something to the 'hangs' section on system health. We were evaluating DeepSee and having massive performance and hang issues. I was trying to do complicated stuff, and the machine was underpowered, but a MAJOR part of the problem was license thrashing. We had four licenses and my queries were essentially trying to spawn multiple processes and hitting the wall, and then trying again and again and again. Not only did we miss this, but it did not occur to support either, until they saw 'This Service Unavailable" messages, and tracked that down to license depletion. Changing our evaluation version to 10 licenses, with no other changes, to the machine, or the queries, instantly took the machine from DOA to usable.

Thank you Sam, this is a good suggestion.  

Thanks for your suggestion. I added that to the table.

I would add something about 'This Service Unavailable"  messages when trying to run Analyzer, Architect etc. - this is not a 'hang', but is symptomatic of no more available licenses. Sorry I didn't mention this explicitly before. Even the support guys on the phone with me did not know that the system was trying to tell us it had run out of licenses - they had to check the logs. Until a more meaningful error message is provided, perhaps this should be added to the chart too.

 

Cheers. 

Journaling the DeepSee cache (the ^DeepSee.Cache* globals) can also cause low query performance

Journaling a DeepSee environment is a real performance killer.
Take a look of the Journal when  using a Bitmap Index 

enlightened The article is considered as InterSystems Data Platform Best Practice.