[23:53:04.796] src/tools/ci/cores_backtrace.sh linux /tmp/cores [23:53:05.510] dumping /tmp/cores/postgres-3-33677.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [23:53:07.394] [New LWP 33677] [23:53:07.394] [New LWP 32582] [23:53:07.394] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [23:53:08.165] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [23:53:09.116] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [23:53:10.392] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [23:53:12.957] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [23:53:12.957] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [23:53:15.057] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [23:53:16.272] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [23:53:17.894] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [23:53:18.977] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [23:53:19.757] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [23:53:21.060] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [23:53:22.741] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [23:53:24.185] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [23:53:25.097] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [23:53:26.634] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [23:53:28.175] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [23:53:28.646] [Thread debugging using libthread_db enabled] [23:53:28.646] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [23:53:29.876] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [23:53:31.308] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [23:53:32.372] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [23:53:32.995] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [23:53:33.851] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [23:53:36.322] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [23:53:37.259] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [23:53:38.058] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [23:53:39.347] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [23:53:40.287] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [23:53:41.950] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [23:53:43.029] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [23:53:43.805] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [23:53:45.236] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [23:53:45.860] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [23:53:47.389] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [23:53:48.150] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [23:53:49.377] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [23:53:50.164] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [23:53:51.251] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [23:53:52.199] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [23:53:53.453] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [23:53:54.076] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [23:53:55.010] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [23:53:55.797] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [23:53:56.296] Core was generated by `postgres: standby2: startup recovering 000000010000000000000006 '. [23:53:56.296] Program terminated with signal SIGQUIT, Quit. [23:53:56.296] #0 0x00007fc988108fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.325] [Current thread is 1 (LWP 33677)] [23:53:56.434] [23:53:56.434] Thread 2 (Thread 0x7fc985a9fa40 (LWP 32582)): [23:53:56.434] #0 0x00007fc9881cba61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.434] No symbol table info available. [23:53:56.434] #1 0x00007fc9881b9e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.434] No symbol table info available. [23:53:56.434] #2 0x00007fc9881ba497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.434] No symbol table info available. [23:53:56.434] #3 0x00007fc9881b9cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.434] No symbol table info available. [23:53:56.434] #4 0x00007fc988115a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.434] No symbol table info available. [23:53:56.434] #5 0x0000563ba6b0be0e in RestoreArchivedFile (path=path@entry=0x7ffcd631ef70 "", xlogfname=xlogfname@entry=0x7ffcd631ef30 "00000002.history", recovername=recovername@entry=0x563ba75fed50 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [23:53:56.434] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\066\000\000\000\000\000\000\000\000\365\335E\247\000\000\000\062\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\004\265_\247;V\000\000\b\000\000\000\000\000\000\000\331Y_\247;V\000\000X\356\061\326\374\177\000\000\325Y_\247;V\000\000!\331E\247;V\000\000\341Y_\247;V\000\000\331Y_\247;V\000\000X\356\061\326\374\177\000\000\325Y_\247;V\000\000\030\371\251\205\311\177\000\000\277\342E\247;V", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000"... [23:53:56.434] xlogRestoreCmd = 0x563ba8cc0608 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/040_standby_failover_slots_sync/data/t_040_standby_failover_slots_sync_publisher_data/archives/00000002.history\" \"pg_wal/RECOVERYHISTORY\"" [23:53:56.434] lastRestartPointFname = '0' , "\000\355_\247;V\000\000\370\351\061\326\374\177\000\000\003\355_\247;V\000\000\030\371\251\205\311\177\000\000}\343E\247;V", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [23:53:56.434] rc = [23:53:56.434] stat_buf = {st_dev = 140503507663128, st_ino = 208964805501, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 48, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 4673356, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 3891110078048108544, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __glibc_reserved = {8, 94814211126532, 140723902081528}} [23:53:56.434] restartSegNo = [23:53:56.434] restartRedoPtr = 94814209433077 [23:53:56.434] restartTli = 0 [23:53:56.434] __func__ = "RestoreArchivedFile" [23:53:56.434] #6 0x0000563ba6ae20a5 in existsTimeLineHistory (probeTLI=probeTLI@entry=2) at ../src/backend/access/transam/timeline.c:235 [23:53:56.434] path = '\000' , "\001\000\000\000\000\000\000\000\r\255\261\246;V\000\000|\334_\247;V\000\000\a\000\000\000\r\000\000\000w\334_\247;V\000\000\030\371\251\205\311\177\000\000`\362\061\326\374\177\000\000a\353E\247;V\000\000\000\000\000\000\000\000\000\000BB\376\247;V\000\000\300B\376\247;V\000\000$\361\061\326\374\177\000\000\230>k\247;V\000\000QB\376\247;V\000\000\300B\376\247;V\000\000D\361\061\326\374\177\000\000\247>k\247;V\000\000\300B\376\247;V\000\000\002\000\000\000\000\000\000\000\036\363E\247;V\000\000F\361\061\326\374\177\000\000D\361\061\326\374\177", '\000' ... [23:53:56.434] histfname = "00000002.history", '\000' [23:53:56.434] fd = [23:53:56.434] __func__ = "existsTimeLineHistory" [23:53:56.434] #7 0x0000563ba6ae2190 in findNewestTimeLine (startTLI=startTLI@entry=1) at ../src/backend/access/transam/timeline.c:277 [23:53:56.434] newestTLI = 1 [23:53:56.434] probeTLI = 2 [23:53:56.434] #8 0x0000563ba6b16dd9 in rescanLatestTimeLine (replayTLI=replayTLI@entry=1, replayLSN=replayLSN@entry=100683512) at ../src/backend/access/transam/xlogrecovery.c:4124 [23:53:56.434] newExpectedTLEs = [23:53:56.434] found = [23:53:56.434] cell = [23:53:56.434] newtarget = [23:53:56.434] oldtarget = 1 [23:53:56.434] currentTle = 0x0 [23:53:56.434] __func__ = "rescanLatestTimeLine" [23:53:56.434] #9 0x0000563ba6b1bf89 in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=100683536, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=100683512, replayTLI=1, replayLSN=100683512, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3685 [23:53:56.434] oldSource = XLOG_FROM_STREAM [23:53:56.434] startWalReceiver = false [23:53:56.434] last_fail_time = 765417045517805 [23:53:56.434] now = [23:53:56.434] streaming_reply_sent = true [23:53:56.434] __func__ = "WaitForWALToBecomeAvailable" [23:53:56.434] #10 0x0000563ba6b1c9bf in XLogPageRead (xlogreader=0x563ba8c6df50, targetPagePtr=100679680, reqLen=3856, targetRecPtr=100683512, readBuf=0x563ba8cbcdd8 "\025\321\006") at ../src/backend/access/transam/xlogrecovery.c:3362 [23:53:56.434] private = 0x563ba8c6df30 [23:53:56.434] emode = 15 [23:53:56.434] targetPageOff = 16384 [23:53:56.434] targetSegNo = 6 [23:53:56.434] r = [23:53:56.434] __func__ = "XLogPageRead" [23:53:56.434] #11 0x0000563ba6b24f53 in ReadPageInternal (state=state@entry=0x563ba8c6df50, pageptr=pageptr@entry=100679680, reqLen=reqLen@entry=3856) at ../src/backend/access/transam/xlogreader.c:1060 [23:53:56.434] readLen = [23:53:56.434] targetPageOff = 16384 [23:53:56.434] targetSegNo = 6 [23:53:56.434] hdr = [23:53:56.434] #12 0x0000563ba6b26bae in XLogDecodeNextRecord (state=state@entry=0x563ba8c6df50, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [23:53:56.434] RecPtr = 100683512 [23:53:56.434] record = [23:53:56.434] targetPagePtr = 100679680 [23:53:56.434] randAccess = false [23:53:56.434] len = [23:53:56.434] total_len = [23:53:56.434] targetRecOff = 3832 [23:53:56.434] pageHeaderSize = [23:53:56.434] assembled = false [23:53:56.434] gotheader = [23:53:56.434] readOff = [23:53:56.434] decoded = 0x0 [23:53:56.434] errormsg = 0x563ba76a2241 "LOG" [23:53:56.434] #13 0x0000563ba6b27750 in XLogReadAhead (state=0x563ba8c6df50, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [23:53:56.434] result = [23:53:56.434] #14 0x0000563ba6b11f59 in XLogPrefetcherNextBlock (pgsr_private=94814234668416, lsn=0x563ba8cd2758) at ../src/backend/access/transam/xlogprefetcher.c:492 [23:53:56.434] nonblocking = [23:53:56.434] record = [23:53:56.434] prefetcher = 0x563ba8c6ed80 [23:53:56.434] reader = 0x563ba8c6df50 [23:53:56.434] replaying_lsn = 100683464 [23:53:56.434] __func__ = "XLogPrefetcherNextBlock" [23:53:56.434] #15 0x0000563ba6b16322 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [23:53:56.434] No locals. [23:53:56.434] #16 lrq_complete_lsn (lsn=0, lrq=0x563ba8cd2688) at ../src/backend/access/transam/xlogprefetcher.c:290 [23:53:56.434] No locals. [23:53:56.434] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x563ba8c6ed80, errmsg=errmsg@entry=0x7ffcd63236b8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [23:53:56.434] record = [23:53:56.434] replayed_up_to = 0 [23:53:56.434] #18 0x0000563ba6b1d1ec in ReadRecord (xlogprefetcher=0x563ba8c6ed80, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=1) at ../src/backend/access/transam/xlogrecovery.c:3162 [23:53:56.434] errormsg = 0x563ba8c6e760 "" [23:53:56.434] record = [23:53:56.434] xlogreader = 0x563ba8c6df50 [23:53:56.434] private = [23:53:56.434] __func__ = "ReadRecord" [23:53:56.434] #19 0x0000563ba6b20410 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1843 [23:53:56.434] xtime = [23:53:56.434] ru0 = {tv = {tv_sec = 1712101840, tv_usec = 55917}, ru = {ru_utime = {tv_sec = 0, tv_usec = 1526}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 16076, __ru_maxrss_word = 16076}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 404, __ru_minflt_word = 404}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 4, __ru_nvcsw_word = 4}, {ru_nivcsw = 2, __ru_nivcsw_word = 2}}} [23:53:56.434] record = 0x7fc985359f70 [23:53:56.434] reachedRecoveryTarget = false [23:53:56.434] replayTLI = 1 [23:53:56.434] __func__ = "PerformWalRecovery" [23:53:56.453] #20 0x0000563ba6b0a026 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5724 [23:53:56.453] Insert = [23:53:56.453] checkPoint = {redo = 83886120, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 750}, nextOid = 24576, nextMulti = 1, nextMultiOffset = 0, oldestXid = 731, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1712101838, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 750} [23:53:56.453] wasShutdown = false [23:53:56.453] didCrash = [23:53:56.453] haveTblspcMap = false [23:53:56.453] haveBackupLabel = true [23:53:56.453] EndOfLog = [23:53:56.453] EndOfLogTLI = [23:53:56.453] newTLI = [23:53:56.453] performedWalRecovery = [23:53:56.453] endOfRecoveryInfo = [23:53:56.453] abortedRecPtr = [23:53:56.453] missingContrecPtr = [23:53:56.453] oldestActiveXID = [23:53:56.453] promoted = false [23:53:56.453] __func__ = "StartupXLOG" [23:53:56.453] #21 0x0000563ba6fc32c8 in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [23:53:56.453] No locals. [23:53:56.453] #22 0x0000563ba6fbaa92 in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:265 [23:53:56.453] pid = [23:53:56.453] #23 0x0000563ba6fbc95d in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [23:53:56.453] pid = [23:53:56.453] __func__ = "StartChildProcess" [23:53:56.453] #24 0x0000563ba6fc2a27 in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x563ba8c6ce10) at ../src/backend/postmaster/postmaster.c:1364 [23:53:56.453] opt = [23:53:56.453] status = [23:53:56.453] userDoption = [23:53:56.453] listen_addr_saved = false [23:53:56.453] output_config_variable = [23:53:56.453] __func__ = "PostmasterMain" [23:53:56.453] #25 0x0000563ba6dea9ca in main (argc=4, argv=0x563ba8c6ce10) at ../src/backend/main/main.c:197 [23:53:56.453] do_check_root = [23:53:56.453] [23:53:56.453] Thread 1 (LWP 33677): [23:53:56.453] #0 0x00007fc988108fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.453] No symbol table info available. [23:53:56.453] #1 0x00007fc9881ba13a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.453] No symbol table info available. [23:53:56.453] #2 0x00007fc9881cba6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [23:53:56.453] No symbol table info available.