I recently faced disk space issue on SharePoint Fast 2010 Crawl server. We need to find out what's consuming the space. If it is the index or any core components we don't have an option other than increasing the space.

Investigate DiskSpace

Let's begin the journey with identifying the files, we will use a simple tool (windirstat) to find the space consuming files.

The initial results were showing that the core components are not the issue and all that's consuming is the logs. I was surprised to see a SPRel log that was 27GB and growing in standard phase, Next big thing was query logs. we have a PowerShell script that cleans up the query logs that are older than 60 days.

1. FASTSEARCH\var\log\querylogs\  (Safely can be cleaned up)
2. FASTSEARCH\var\log\sprel\
3. FASTSEARCH\data\sprel\worker

We realized something is going wrong with the SPRel process. Just to give you an idea of the SPRel service.

"SPRel is a clickthrough log analysis engine that improves search results relevancy by analyzing the entries that users click on in search result sets. In FAST Search server 2010, we can use spreladmin.exe to configure SPRel, to schedule the clickthrough log analyses, and to retrieve status information"

Investigate SPRel

Check if the service is running as expected. Command to check the status should be executed in command prompt

<drive>:FastSearch\Bin\spreladmin showstatus

The process is actually stuck and not working as expected. This has stopped working in 2013 and till now in 2016. I think we are in the right direction. Now we have to analyze the log to find the reasons. Here comes another problem log is too huge to open it in notepad(27GB). It's a Handy tool that saved my life Large Text File Viewer no installation required doesn't crash your CPU it lazy loads the file with continuous query mode.

Logs

The possible failures found in the log, Disk Space

Disk Space wasn't enough to process

Impossible to schedule more targets (timeout 60s). 4 ready targets:

make_1032.urls_by_urlid.1_b1. Target was never started. Too large output for the workers. Input size : 153196411

make_1032.urls_by_urlid.3_b3. Target was never started. Too large output for the workers. Input size : 153952094

make_1032.urls_by_urlid.0_b0. Target was never started. Too large output for the workers. Input size : 153724399

make_1032.urls_by_urlid.2_b2. Target was never started. Too large output for the workers. Input size : 153472337

--- stopping build.

The Server didn't respond and it has failed.

systemmsg Reset of analysis completed.

systemmsg Analysis failed 10 times, will retry from start.

systemmsg sprelrsclient failed. stdout: Connected to ResourceStore stderr: Could not list resources.. The remote server returned an error: (500) Internal Server Error.

I will take some time to figure out the solution and will discuss in the next post.