[01:19:52.873] src/tools/ci/cores_backtrace.sh linux /tmp/cores [01:19:53.677] dumping /tmp/cores/postgres-3-34774.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [01:19:55.815] [New LWP 34774] [01:19:55.815] [New LWP 33977] [01:19:55.815] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [01:19:56.574] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [01:19:57.760] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [01:19:59.193] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [01:20:02.102] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [01:20:02.102] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [01:20:04.451] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [01:20:05.660] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [01:20:07.403] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [01:20:08.521] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [01:20:09.364] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [01:20:10.760] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [01:20:12.554] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [01:20:14.017] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [01:20:14.926] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [01:20:16.708] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [01:20:18.687] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [01:20:19.156] [Thread debugging using libthread_db enabled] [01:20:19.156] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [01:20:20.599] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [01:20:22.130] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [01:20:23.342] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [01:20:24.037] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [01:20:24.954] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [01:20:27.794] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [01:20:28.764] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [01:20:29.620] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [01:20:31.437] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [01:20:32.401] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [01:20:34.260] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [01:20:35.373] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [01:20:36.191] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [01:20:37.662] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [01:20:38.306] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [01:20:40.122] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [01:20:40.882] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [01:20:42.103] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [01:20:43.011] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [01:20:44.166] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [01:20:45.229] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [01:20:46.531] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [01:20:47.241] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [01:20:48.237] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [01:20:49.055] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [01:20:49.551] Core was generated by `postgres: standby1: startup recovering 000000010000000000000004 '. [01:20:49.551] Program terminated with signal SIGQUIT, Quit. [01:20:49.551] #0 0x00007fd33cc58fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.580] [Current thread is 1 (LWP 34774)] [01:20:49.713] [01:20:49.713] Thread 2 (Thread 0x7fd33a5efa40 (LWP 33977)): [01:20:49.713] #0 0x00007fd33cd1ba61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.713] No symbol table info available. [01:20:49.713] #1 0x00007fd33cd09e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.713] No symbol table info available. [01:20:49.713] #2 0x00007fd33cd0a497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.713] No symbol table info available. [01:20:49.713] #3 0x00007fd33cd09cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.713] No symbol table info available. [01:20:49.713] #4 0x00007fd33cc65a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.713] No symbol table info available. [01:20:49.713] #5 0x000056448ef2990b in RestoreArchivedFile (path=path@entry=0x7ffca8ae2780 "", xlogfname=xlogfname@entry=0x7ffca8ae2740 "00000002.history", recovername=recovername@entry=0x56448f9f90f0 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:178 [01:20:49.713] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\247S\333<\323\000\000\064\000\000\000\000\000\000\000\000\000\224{D\000\000\000\062\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\304X\237\217DV\000\000\b\000\000\000\000\000\000\000I\000\237\217DV\000\000h&\256\250\374\177\000\000E\000\237\217DV\000\000\310\364\205\217DV\000\000Q\000\237\217DV\000\000I\000\237\217DV\000\000h&\256\250\374\177\000\000E\000\237\217DV\000\000\030\371^:\323\177\000\000f\376\205\217DV", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [01:20:49.713] xlogRestoreCmd = 0x5644914a1918 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/040_standby_failover_slots_sync/data/t_040_standby_failover_slots_sync_publisher_data/archives/00000002.history\" \"pg_wal/RECOVERYHISTORY\"" [01:20:49.713] lastRestartPointFname = '0' , "\000\220\237\217DV\000\000\b\"\256\250\374\177\000\000\243\220\237\217DV\000\000\030\371^:\323\177\000\000$\377\205\217DV", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' , "/tmp/cirr"... [01:20:49.713] rc = [01:20:49.713] stat_buf = {st_dev = 140545194129688, st_ino = 208566353700, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 48, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 4673356, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 3746994889972252672, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __glibc_reserved = {8, 94852467349700, 140723138470408}} [01:20:49.713] restartSegNo = [01:20:49.713] restartRedoPtr = 94852465686940 [01:20:49.713] restartTli = 0 [01:20:49.713] __func__ = "RestoreArchivedFile" [01:20:49.713] #6 0x000056448eeffcfc in existsTimeLineHistory (probeTLI=probeTLI@entry=2) at ../src/backend/access/transam/timeline.c:235 [01:20:49.713] path = '\000' , "\001\000\000\000\000\000\000\000\n\210\363\216DV\000\000\001\000\000\000\000\000\000\000\006\000\000\000\r", '\000' , "\030\371^:\323\177\000\000p*\256\250\374\177\000\000\b\a\206\217DV\000\000\000\000\000\000\000\000\000\000\240\037\000\000\377\377\002\000\300\020:\220DV\000\000\064)\256\250\374\177\000\000\270x\252\217DV\000\000Q\020:\220DV\000\000\300\020:\220DV\000\000T)\256\250\374\177\000\000\307x\252\217DV\000\000\300\020:\220DV\000\000\002\000\000\000\000\000\000\000\305\016\206\217DV\000\000V)\256\250\374\177\000\000T)\256\250\374\177", '\000' ... [01:20:49.713] histfname = "00000002.history", '\000' [01:20:49.713] fd = [01:20:49.713] __func__ = "existsTimeLineHistory" [01:20:49.713] #7 0x000056448eeffde7 in findNewestTimeLine (startTLI=startTLI@entry=1) at ../src/backend/access/transam/timeline.c:277 [01:20:49.713] newestTLI = 1 [01:20:49.713] probeTLI = 2 [01:20:49.713] #8 0x000056448ef348d6 in rescanLatestTimeLine (replayTLI=replayTLI@entry=1, replayLSN=replayLSN@entry=67109168) at ../src/backend/access/transam/xlogrecovery.c:4095 [01:20:49.713] newExpectedTLEs = [01:20:49.713] found = [01:20:49.713] cell = [01:20:49.713] newtarget = [01:20:49.713] oldtarget = 1 [01:20:49.713] currentTle = 0x0 [01:20:49.713] __func__ = "rescanLatestTimeLine" [01:20:49.713] #9 0x000056448ef39a86 in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67109192, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67109168, replayTLI=1, replayLSN=67109168, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3656 [01:20:49.713] oldSource = XLOG_FROM_STREAM [01:20:49.713] startWalReceiver = false [01:20:49.713] last_fail_time = 761707043741389 [01:20:49.713] now = [01:20:49.713] streaming_reply_sent = true [01:20:49.713] __func__ = "WaitForWALToBecomeAvailable" [01:20:49.713] #10 0x000056448ef3a4bc in XLogPageRead (xlogreader=0x56449144ef70, targetPagePtr=67108864, reqLen=328, targetRecPtr=67109168, readBuf=0x56449149e0e8 "\024\321\006") at ../src/backend/access/transam/xlogrecovery.c:3333 [01:20:49.713] private = 0x56449144ef50 [01:20:49.713] emode = 15 [01:20:49.713] targetPageOff = 0 [01:20:49.713] targetSegNo = 4 [01:20:49.713] r = [01:20:49.713] __func__ = "XLogPageRead" [01:20:49.713] #11 0x000056448ef428dc in ReadPageInternal (state=state@entry=0x56449144ef70, pageptr=pageptr@entry=67108864, reqLen=reqLen@entry=328) at ../src/backend/access/transam/xlogreader.c:1062 [01:20:49.713] readLen = [01:20:49.713] targetPageOff = 0 [01:20:49.713] targetSegNo = 4 [01:20:49.713] hdr = [01:20:49.713] #12 0x000056448ef44537 in XLogDecodeNextRecord (state=state@entry=0x56449144ef70, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:599 [01:20:49.713] RecPtr = 67109168 [01:20:49.713] record = [01:20:49.713] targetPagePtr = 67108864 [01:20:49.713] randAccess = false [01:20:49.713] len = [01:20:49.713] total_len = [01:20:49.713] targetRecOff = 304 [01:20:49.713] pageHeaderSize = [01:20:49.713] assembled = false [01:20:49.713] gotheader = [01:20:49.713] readOff = [01:20:49.713] decoded = 0x0 [01:20:49.713] errormsg = 0x56448fa95e47 "LOG" [01:20:49.713] #13 0x000056448ef450d9 in XLogReadAhead (state=0x56449144ef70, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:975 [01:20:49.713] result = [01:20:49.713] #14 0x000056448ef2fa56 in XLogPrefetcherNextBlock (pgsr_private=94852494982560, lsn=0x5644914b39c8) at ../src/backend/access/transam/xlogprefetcher.c:496 [01:20:49.713] nonblocking = [01:20:49.713] record = [01:20:49.713] prefetcher = 0x56449144fda0 [01:20:49.713] reader = 0x56449144ef70 [01:20:49.713] replaying_lsn = 67109112 [01:20:49.713] __func__ = "XLogPrefetcherNextBlock" [01:20:49.713] #15 0x000056448ef33e1f in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:256 [01:20:49.713] No locals. [01:20:49.713] #16 lrq_complete_lsn (lsn=0, lrq=0x5644914b3998) at ../src/backend/access/transam/xlogprefetcher.c:294 [01:20:49.713] No locals. [01:20:49.713] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x56449144fda0, errmsg=errmsg@entry=0x7ffca8ae6ec8) at ../src/backend/access/transam/xlogprefetcher.c:1041 [01:20:49.713] record = [01:20:49.713] replayed_up_to = 0 [01:20:49.713] #18 0x000056448ef3ace9 in ReadRecord (xlogprefetcher=0x56449144fda0, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=1) at ../src/backend/access/transam/xlogrecovery.c:3133 [01:20:49.713] errormsg = 0x56449144f780 "" [01:20:49.713] record = [01:20:49.713] xlogreader = 0x56449144ef70 [01:20:49.713] private = [01:20:49.713] __func__ = "ReadRecord" [01:20:49.713] #19 0x000056448ef3de45 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1814 [01:20:49.713] xtime = [01:20:49.713] ru0 = {tv = {tv_sec = 1708391840, tv_usec = 341992}, ru = {ru_utime = {tv_sec = 0, tv_usec = 1305}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 16112, __ru_maxrss_word = 16112}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 407, __ru_minflt_word = 407}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 4, __ru_nvcsw_word = 4}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [01:20:49.713] record = 0x7fd339f33070 [01:20:49.713] reachedRecoveryTarget = false [01:20:49.713] replayTLI = 1 [01:20:49.735] __func__ = "PerformWalRecovery" [01:20:49.735] #20 0x000056448ef27b94 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5733 [01:20:49.735] Insert = [01:20:49.735] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 739}, nextOid = 24576, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1708391839, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 739} [01:20:49.735] wasShutdown = false [01:20:49.735] didCrash = [01:20:49.735] haveTblspcMap = false [01:20:49.735] haveBackupLabel = true [01:20:49.735] EndOfLog = [01:20:49.735] EndOfLogTLI = [01:20:49.735] newTLI = [01:20:49.735] performedWalRecovery = [01:20:49.735] endOfRecoveryInfo = [01:20:49.735] abortedRecPtr = [01:20:49.735] missingContrecPtr = [01:20:49.735] oldestActiveXID = [01:20:49.735] promoted = false [01:20:49.735] __func__ = "StartupXLOG" [01:20:49.735] #21 0x000056448f3d85ae in StartupProcessMain () at ../src/backend/postmaster/startup.c:254 [01:20:49.735] No locals. [01:20:49.735] #22 0x000056448f3c8a14 in AuxiliaryProcessMain (auxtype=auxtype@entry=StartupProcess) at ../src/backend/postmaster/auxprocess.c:142 [01:20:49.735] __func__ = "AuxiliaryProcessMain" [01:20:49.735] #23 0x000056448f3d38fa in StartChildProcess (type=type@entry=StartupProcess) at ../src/backend/postmaster/postmaster.c:5311 [01:20:49.735] pid = [01:20:49.735] __func__ = "StartChildProcess" [01:20:49.735] #24 0x000056448f3d7cee in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x56449144dde0) at ../src/backend/postmaster/postmaster.c:1459 [01:20:49.735] opt = [01:20:49.735] status = [01:20:49.735] userDoption = [01:20:49.735] listen_addr_saved = false [01:20:49.735] output_config_variable = [01:20:49.735] __func__ = "PostmasterMain" [01:20:49.735] #25 0x000056448f20284b in main (argc=4, argv=0x56449144dde0) at ../src/backend/main/main.c:198 [01:20:49.735] do_check_root = [01:20:49.735] [01:20:49.735] Thread 1 (LWP 34774): [01:20:49.735] #0 0x00007fd33cc58fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.735] No symbol table info available. [01:20:49.735] #1 0x00007fd33cd0a13a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.735] No symbol table info available. [01:20:49.735] #2 0x00007fd33cd1ba6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [01:20:49.735] No symbol table info available. [01:20:49.755] -e [01:20:49.755] [01:20:49.755] [01:20:50.278] dumping /tmp/cores/sh-3-30166.core for /bin/sh [01:20:51.342] Downloading separate debug info for /bin/sh... [01:20:51.809] [New LWP 30166] [01:20:51.810] Core was generated by `sh -c cp "/tmp/cirrus-ci-build/build/testrun/recovery/020_archive_status/data/t'. [01:20:51.810] Program terminated with signal SIGQUIT, Quit. [01:20:51.810] #0 0x00007fd98d1d3090 in ?? () [01:20:51.810] [01:20:51.810] Thread 1 (LWP 30166): [01:20:51.810] #0 0x00007fd98d1d3090 in ?? () [01:20:51.810] No symbol table info available. [01:20:51.810] #1 0x0000000000000003 in ?? () [01:20:51.810] No symbol table info available. [01:20:51.810] #2 0x00007ffc277d1498 in ?? () [01:20:51.810] No symbol table info available. [01:20:51.810] #3 0x00007ffc277d149b in ?? () [01:20:51.810] No symbol table info available. [01:20:51.810] #4 0x00007ffc277d149e in ?? () [01:20:51.810] No symbol table info available. [01:20:51.810] #5 0x0000000000000000 in ?? () [01:20:51.810] No symbol table info available.