[01:13:24.026] src/tools/ci/cores_backtrace.sh linux /tmp/cores [01:13:24.739] dumping /tmp/cores/postgres-3-25762.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [01:13:26.543] [New LWP 25762] [01:13:26.543] [New LWP 24557] [01:13:26.543] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [01:13:27.302] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [01:13:28.192] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [01:13:29.475] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [01:13:32.007] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [01:13:32.007] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [01:13:34.054] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [01:13:35.265] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [01:13:36.918] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [01:13:37.999] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [01:13:38.785] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [01:13:40.091] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [01:13:41.781] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [01:13:43.222] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [01:13:44.132] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [01:13:45.668] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [01:13:47.233] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [01:13:47.706] [Thread debugging using libthread_db enabled] [01:13:47.706] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [01:13:48.773] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [01:13:50.207] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [01:13:51.264] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [01:13:51.888] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [01:13:52.745] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [01:13:55.244] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [01:13:56.175] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [01:13:56.970] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [01:13:58.261] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [01:13:59.194] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [01:14:00.858] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [01:14:01.944] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [01:14:02.727] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [01:14:04.144] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [01:14:04.771] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [01:14:06.299] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [01:14:07.060] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [01:14:08.285] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [01:14:09.069] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [01:14:10.176] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [01:14:11.123] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [01:14:12.366] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [01:14:12.994] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [01:14:13.944] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [01:14:14.717] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [01:14:15.226] Core was generated by `postgres: standby2: startup recovering 000000010000000000000006 '. [01:14:15.226] Program terminated with signal SIGQUIT, Quit. [01:14:15.226] #0 0x00007f19d1740fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.256] [Current thread is 1 (LWP 25762)] [01:14:15.391] [01:14:15.391] Thread 2 (Thread 0x7f19cf0d7a40 (LWP 24557)): [01:14:15.391] #0 0x00007f19d1803a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.391] No symbol table info available. [01:14:15.391] #1 0x00007f19d17f1e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.391] No symbol table info available. [01:14:15.391] #2 0x00007f19d17f2497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.391] No symbol table info available. [01:14:15.391] #3 0x00007f19d17f1cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.391] No symbol table info available. [01:14:15.391] #4 0x00007f19d174da69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.391] No symbol table info available. [01:14:15.391] #5 0x0000559e180844c6 in RestoreArchivedFile (path=path@entry=0x7fff49b7b8f0 "", xlogfname=xlogfname@entry=0x7fff49b7b8b0 "00000002.history", recovername=recovername@entry=0x559e18b959f0 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [01:14:15.391] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\066\000\000\000\000\000\000\000\000l\261\236\030\000\000\000\062\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\314!\271\030\236U\000\000\b\000\000\000\000\000\000\000\271Ǹ\030\236U\000\000ط\267I\377\177\000\000\265Ǹ\030\236U\000\000\230\254\236\030\236U\000\000\301Ǹ\030\236U\000\000\271Ǹ\030\236U\000\000ط\267I\377\177\000\000\265Ǹ\030\236U\000\000\030y\r\317\031\177\000\000\066\266\236\030\236U", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000\031\000\000\000"... [01:14:15.391] xlogRestoreCmd = 0x559e1b248648 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/040_standby_failover_slots_sync/data/t_040_standby_failover_slots_sync_publisher_data/archives/00000002.history\" \"pg_wal/RECOVERYHISTORY\"" [01:14:15.391] lastRestartPointFname = '0' , "\000Y\271\030\236U\000\000x\263\267I\377\177\000\000\243Y\271\030\236U\000\000\030y\r\317\031\177\000\000\364\266\236\030\236U", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [01:14:15.391] rc = [01:14:15.391] stat_buf = {st_dev = 139748824676632, st_ino = 206571484916, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 48, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 4673356, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 3891110078048108544, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __glibc_reserved = {8, 94137507979724, 140734430163832}} [01:14:15.391] restartSegNo = [01:14:15.391] restartRedoPtr = 94137506247020 [01:14:15.391] restartTli = 0 [01:14:15.391] __func__ = "RestoreArchivedFile" [01:14:15.391] #6 0x0000559e18058eaf in existsTimeLineHistory (probeTLI=probeTLI@entry=2) at ../src/backend/access/transam/timeline.c:235 [01:14:15.391] path = '\000' , "\001\000\000\000\000\000\000\000\305\063\t\030\236U\000\000 \275\267I\377\177\000\000\a\000\000\000\r\000\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "ؾ\236\030\236U\000\000\000\000\000\000\000\000\000\000\342FY\031\236U\000\000`GY\031\236U\000\000\240\037\000\000\377\377\002\000\070\302\304\030\236U\000\000\361FY\031\236U\000\000`GY\031\236U\000\000ĺ\267I\377\177\000\000G\302\304\030\236U\000\000`GY\031\236U\000\000\002\000\000\000\000\000\000\000\225ƞ\030\236U\000\000ƺ\267I\377\177\000\000ĺ\267I\377\177", '\000' ... [01:14:15.391] histfname = "00000002.history", '\000' [01:14:15.391] fd = [01:14:15.391] __func__ = "existsTimeLineHistory" [01:14:15.391] #7 0x0000559e18058f9a in findNewestTimeLine (startTLI=startTLI@entry=1) at ../src/backend/access/transam/timeline.c:277 [01:14:15.391] newestTLI = 1 [01:14:15.391] probeTLI = 2 [01:14:15.391] #8 0x0000559e1808f491 in rescanLatestTimeLine (replayTLI=replayTLI@entry=1, replayLSN=replayLSN@entry=100683512) at ../src/backend/access/transam/xlogrecovery.c:4113 [01:14:15.391] newExpectedTLEs = [01:14:15.391] found = [01:14:15.391] cell = [01:14:15.391] newtarget = [01:14:15.391] oldtarget = 1 [01:14:15.391] currentTle = 0x0 [01:14:15.391] __func__ = "rescanLatestTimeLine" [01:14:15.391] #9 0x0000559e18094641 in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=100683536, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=100683512, replayTLI=1, replayLSN=100683512, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3674 [01:14:15.391] oldSource = XLOG_FROM_STREAM [01:14:15.391] startWalReceiver = false [01:14:15.391] last_fail_time = 768273053474529 [01:14:15.391] now = [01:14:15.391] streaming_reply_sent = true [01:14:15.391] __func__ = "WaitForWALToBecomeAvailable" [01:14:15.391] #10 0x0000559e18095077 in XLogPageRead (xlogreader=0x559e1b1f5f90, targetPagePtr=100679680, reqLen=3856, targetRecPtr=100683512, readBuf=0x559e1b244e18 "\025\321\006") at ../src/backend/access/transam/xlogrecovery.c:3351 [01:14:15.391] private = 0x559e1b1f5f70 [01:14:15.391] emode = 15 [01:14:15.391] targetPageOff = 16384 [01:14:15.391] targetSegNo = 6 [01:14:15.391] r = [01:14:15.391] __func__ = "XLogPageRead" [01:14:15.391] #11 0x0000559e1809d4d2 in ReadPageInternal (state=state@entry=0x559e1b1f5f90, pageptr=pageptr@entry=100679680, reqLen=reqLen@entry=3856) at ../src/backend/access/transam/xlogreader.c:1060 [01:14:15.391] readLen = [01:14:15.391] targetPageOff = 16384 [01:14:15.391] targetSegNo = 6 [01:14:15.391] hdr = [01:14:15.391] #12 0x0000559e1809f12d in XLogDecodeNextRecord (state=state@entry=0x559e1b1f5f90, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [01:14:15.391] RecPtr = 100683512 [01:14:15.391] record = [01:14:15.391] targetPagePtr = 100679680 [01:14:15.391] randAccess = false [01:14:15.391] len = [01:14:15.391] total_len = [01:14:15.391] targetRecOff = 3832 [01:14:15.391] pageHeaderSize = [01:14:15.391] assembled = false [01:14:15.391] gotheader = [01:14:15.391] readOff = [01:14:15.391] decoded = 0x0 [01:14:15.391] errormsg = 0x559e18c3a581 "LOG" [01:14:15.391] #13 0x0000559e1809fccf in XLogReadAhead (state=0x559e1b1f5f90, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [01:14:15.391] result = [01:14:15.391] #14 0x0000559e1808a611 in XLogPrefetcherNextBlock (pgsr_private=94137548238272, lsn=0x559e1b25a798) at ../src/backend/access/transam/xlogprefetcher.c:492 [01:14:15.391] nonblocking = [01:14:15.391] record = [01:14:15.391] prefetcher = 0x559e1b1f6dc0 [01:14:15.391] reader = 0x559e1b1f5f90 [01:14:15.391] replaying_lsn = 100683464 [01:14:15.391] __func__ = "XLogPrefetcherNextBlock" [01:14:15.391] #15 0x0000559e1808e9da in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [01:14:15.391] No locals. [01:14:15.391] #16 lrq_complete_lsn (lsn=0, lrq=0x559e1b25a6c8) at ../src/backend/access/transam/xlogprefetcher.c:290 [01:14:15.391] No locals. [01:14:15.391] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x559e1b1f6dc0, errmsg=errmsg@entry=0x7fff49b80038) at ../src/backend/access/transam/xlogprefetcher.c:1037 [01:14:15.391] record = [01:14:15.391] replayed_up_to = 0 [01:14:15.391] #18 0x0000559e180958a4 in ReadRecord (xlogprefetcher=0x559e1b1f6dc0, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=1) at ../src/backend/access/transam/xlogrecovery.c:3151 [01:14:15.391] errormsg = 0x559e1b1f67a0 "" [01:14:15.391] record = [01:14:15.391] xlogreader = 0x559e1b1f5f90 [01:14:15.391] private = [01:14:15.391] __func__ = "ReadRecord" [01:14:15.391] #19 0x0000559e18098a3b in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1832 [01:14:15.391] xtime = [01:14:15.391] ru0 = {tv = {tv_sec = 1714957847, tv_usec = 140875}, ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 1378}, {ru_maxrss = 16096, __ru_maxrss_word = 16096}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 400, __ru_minflt_word = 400}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 4, __ru_nvcsw_word = 4}, {ru_nivcsw = 2, __ru_nivcsw_word = 2}}} [01:14:15.391] record = 0x7f19ce991f70 [01:14:15.391] reachedRecoveryTarget = false [01:14:15.411] replayTLI = 1 [01:14:15.411] __func__ = "PerformWalRecovery" [01:14:15.411] #20 0x0000559e1808245e in StartupXLOG () at ../src/backend/access/transam/xlog.c:5806 [01:14:15.411] Insert = [01:14:15.411] checkPoint = {redo = 83886120, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 751}, nextOid = 24576, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1714957846, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 751} [01:14:15.411] wasShutdown = false [01:14:15.411] didCrash = [01:14:15.411] haveTblspcMap = false [01:14:15.411] haveBackupLabel = true [01:14:15.411] EndOfLog = [01:14:15.411] EndOfLogTLI = [01:14:15.411] newTLI = [01:14:15.411] performedWalRecovery = [01:14:15.411] endOfRecoveryInfo = [01:14:15.411] abortedRecPtr = [01:14:15.411] missingContrecPtr = [01:14:15.411] oldestActiveXID = [01:14:15.411] promoted = false [01:14:15.411] __func__ = "StartupXLOG" [01:14:15.411] #21 0x0000559e18549bc9 in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [01:14:15.411] No locals. [01:14:15.411] #22 0x0000559e185411e5 in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:265 [01:14:15.411] pid = [01:14:15.411] #23 0x0000559e1854325e in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [01:14:15.411] pid = [01:14:15.411] __func__ = "StartChildProcess" [01:14:15.411] #24 0x0000559e18549328 in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x559e1b1f4e50) at ../src/backend/postmaster/postmaster.c:1364 [01:14:15.411] opt = [01:14:15.411] status = [01:14:15.411] userDoption = [01:14:15.411] listen_addr_saved = false [01:14:15.411] output_config_variable = [01:14:15.411] __func__ = "PostmasterMain" [01:14:15.411] #25 0x0000559e1836975b in main (argc=4, argv=0x559e1b1f4e50) at ../src/backend/main/main.c:197 [01:14:15.411] do_check_root = [01:14:15.411] [01:14:15.411] Thread 1 (LWP 25762): [01:14:15.411] #0 0x00007f19d1740fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.411] No symbol table info available. [01:14:15.411] #1 0x00007f19d17f213a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.411] No symbol table info available. [01:14:15.411] #2 0x00007f19d1803a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [01:14:15.411] No symbol table info available.