[09:59:26.181] src/tools/ci/cores_backtrace.sh linux /tmp/cores [09:59:26.876] dumping /tmp/cores/postgres-3-29247.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [09:59:29.628] [New LWP 29247] [09:59:29.628] [New LWP 29185] [09:59:29.628] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [09:59:30.390] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [09:59:31.294] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [09:59:32.867] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [09:59:35.431] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [09:59:35.431] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [09:59:37.499] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [09:59:38.715] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [09:59:40.378] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [09:59:41.482] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [09:59:42.283] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [09:59:43.601] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [09:59:45.304] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [09:59:46.795] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [09:59:47.709] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [09:59:49.247] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [09:59:50.784] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [09:59:51.262] [Thread debugging using libthread_db enabled] [09:59:51.262] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [09:59:52.470] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [09:59:53.903] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [09:59:54.964] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [09:59:55.606] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [09:59:56.457] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [09:59:59.028] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [09:59:59.965] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [10:00:00.772] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [10:00:02.040] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [10:00:02.997] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [10:00:04.708] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [10:00:05.815] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [10:00:06.605] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [10:00:08.056] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [10:00:08.728] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [10:00:10.274] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [10:00:11.038] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [10:00:12.263] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [10:00:13.073] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [10:00:14.220] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [10:00:15.182] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [10:00:16.454] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [10:00:17.091] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [10:00:18.053] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [10:00:18.857] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [10:00:19.349] Core was generated by `postgres: standby2: startup waiting for 000000010000000000000003 '. [10:00:19.349] Program terminated with signal SIGQUIT, Quit. [10:00:19.349] #0 0x00007f1546271fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.378] [Current thread is 1 (LWP 29247)] [10:00:19.500] [10:00:19.500] Thread 2 (Thread 0x7f1543c08a40 (LWP 29185)): [10:00:19.500] #0 0x00007f1546334a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.500] No symbol table info available. [10:00:19.500] #1 0x00007f1546322e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.500] No symbol table info available. [10:00:19.500] #2 0x00007f1546323497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.500] No symbol table info available. [10:00:19.500] #3 0x00007f1546322cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.500] No symbol table info available. [10:00:19.500] #4 0x00007f154627ea69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.500] No symbol table info available. [10:00:19.500] #5 0x000055aec07e88fa in RestoreArchivedFile (path=path@entry=0x7ffdea77d990 "\001", xlogfname=xlogfname@entry=0x7ffdea77d950 "00000002.history", recovername=recovername@entry=0x55aec12bc430 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [10:00:19.500] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b", '\000' , "\071\065\022\301\256U\000\062", '\000' , "\b\000\000\000\000\000\000\000Y3+\301\256U\000\000x\330w\352\375\177\000\000U3+\301\256U\000\000e0\022\301\256U\000\000a3+\301\256U\000\000Y3+\301\256U\000\000x\330w\352\375\177\000\000U3+\301\256U\000\000\030\211\300C\025\177\000\000\003:\022\301\256U", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000"... [10:00:19.500] xlogRestoreCmd = 0x55aec2308568 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/020_archive_status/data/t_020_archive_status_primary_data/archives/00000002.history\" \"pg_wal/RECOVERYHISTORY\"" [10:00:19.500] lastRestartPointFname = '0' , "\000B\022\301\256U", '\000' , "\320\323w\352\375\177\000\000\003:\022\301\256U", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [10:00:19.500] rc = [10:00:19.500] stat_buf = {st_dev = 32768, st_ino = 207868321994, st_nlink = 683419667, st_mode = 1709891786, st_uid = 0, st_gid = 519419874, __pad0 = 0, st_rdev = 1709891786, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 4673356, st_atim = {tv_sec = 0, tv_nsec = 3472113003596310384}, st_mtim = {tv_sec = 3688782553318174768, tv_nsec = 3472328296227680304}, st_ctim = {tv_sec = 808464432, tv_nsec = 0}, __glibc_reserved = {8, 94209053527068, 140728537175064}} [10:00:19.500] restartSegNo = [10:00:19.500] restartRedoPtr = 94209051866425 [10:00:19.500] restartTli = 0 [10:00:19.500] __func__ = "RestoreArchivedFile" [10:00:19.500] #6 0x000055aec07bee25 in existsTimeLineHistory (probeTLI=probeTLI@entry=2) at ../src/backend/access/transam/timeline.c:235 [10:00:19.500] path = "\001", '\000' , "\001\000\000\000\000\000\000\000\374y\177\300\256U\000\000\300^+®U\000\000\377\377\377\377\r\000\000\000pg_wal/00000001", '0' , "3\000\000\000\000\000\000\000\000\000h\370@F\025\177\000\000\240\335w\352\375\177\000\000\212F\274G\025\177\000\000H\277\305\301\256U", '\000' , " \000\000\000\000\000\000\070M0®U\000\000\005\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\060", '\000' , "\240\037"... [10:00:19.500] histfname = "00000002.history", '\000' , "\030\255\061®U\000\000\001\000\000\000\000\000\000" [10:00:19.500] fd = [10:00:19.500] __func__ = "existsTimeLineHistory" [10:00:19.500] #7 0x000055aec07bef10 in findNewestTimeLine (startTLI=startTLI@entry=1) at ../src/backend/access/transam/timeline.c:277 [10:00:19.500] newestTLI = 1 [10:00:19.500] probeTLI = 2 [10:00:19.500] #8 0x000055aec07f38c5 in rescanLatestTimeLine (replayTLI=replayTLI@entry=1, replayLSN=replayLSN@entry=50331648) at ../src/backend/access/transam/xlogrecovery.c:4110 [10:00:19.500] newExpectedTLEs = [10:00:19.500] found = [10:00:19.500] cell = [10:00:19.500] newtarget = [10:00:19.500] oldtarget = 1 [10:00:19.500] currentTle = 0x0 [10:00:19.500] __func__ = "rescanLatestTimeLine" [10:00:19.500] #9 0x000055aec07f8a75 in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=50331672, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=50331648, replayTLI=1, replayLSN=50331648, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3671 [10:00:19.500] oldSource = XLOG_FROM_STREAM [10:00:19.500] startWalReceiver = false [10:00:19.500] last_fail_time = 763206986803632 [10:00:19.500] now = [10:00:19.500] streaming_reply_sent = false [10:00:19.500] __func__ = "WaitForWALToBecomeAvailable" [10:00:19.500] #10 0x000055aec07f94ab in XLogPageRead (xlogreader=0x55aec22b5ee0, targetPagePtr=50331648, reqLen=24, targetRecPtr=50331648, readBuf=0x55aec2304d38 "\024\321\005") at ../src/backend/access/transam/xlogrecovery.c:3348 [10:00:19.500] private = 0x55aec22b5ec0 [10:00:19.500] emode = 15 [10:00:19.500] targetPageOff = 0 [10:00:19.500] targetSegNo = 3 [10:00:19.500] r = [10:00:19.500] __func__ = "XLogPageRead" [10:00:19.500] #11 0x000055aec08018d0 in ReadPageInternal (state=state@entry=0x55aec22b5ee0, pageptr=pageptr@entry=50331648, reqLen=reqLen@entry=24) at ../src/backend/access/transam/xlogreader.c:1060 [10:00:19.500] readLen = [10:00:19.500] targetPageOff = 0 [10:00:19.500] targetSegNo = 3 [10:00:19.500] hdr = [10:00:19.500] #12 0x000055aec080352b in XLogDecodeNextRecord (state=state@entry=0x55aec22b5ee0, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [10:00:19.500] RecPtr = 50331648 [10:00:19.500] record = [10:00:19.500] targetPagePtr = 50331648 [10:00:19.500] randAccess = false [10:00:19.500] len = [10:00:19.500] total_len = [10:00:19.500] targetRecOff = 0 [10:00:19.500] pageHeaderSize = [10:00:19.500] assembled = false [10:00:19.500] gotheader = [10:00:19.500] readOff = [10:00:19.500] decoded = 0x0 [10:00:19.500] errormsg = 0x55aec07d3e89 "\204\300\017\205q\001" [10:00:19.500] #13 0x000055aec08040cd in XLogReadAhead (state=0x55aec22b5ee0, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [10:00:19.500] result = [10:00:19.500] #14 0x000055aec07eea45 in XLogPrefetcherNextBlock (pgsr_private=94209070296336, lsn=0x55aec231a6e8) at ../src/backend/access/transam/xlogprefetcher.c:492 [10:00:19.500] nonblocking = [10:00:19.500] record = [10:00:19.500] prefetcher = 0x55aec22b6d10 [10:00:19.500] reader = 0x55aec22b5ee0 [10:00:19.500] replaying_lsn = 33563920 [10:00:19.500] __func__ = "XLogPrefetcherNextBlock" [10:00:19.500] #15 0x000055aec07f2e0e in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [10:00:19.500] No locals. [10:00:19.500] #16 lrq_complete_lsn (lsn=50331648, lrq=0x55aec231a5e8) at ../src/backend/access/transam/xlogprefetcher.c:290 [10:00:19.500] No locals. [10:00:19.500] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x55aec22b6d10, errmsg=errmsg@entry=0x7ffdea7820d8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [10:00:19.500] record = [10:00:19.500] replayed_up_to = 50331648 [10:00:19.500] #18 0x000055aec07f9cd8 in ReadRecord (xlogprefetcher=0x55aec22b6d10, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=1) at ../src/backend/access/transam/xlogrecovery.c:3148 [10:00:19.500] errormsg = 0x55aec0ce4c64 "AUATUSH\203\354\bH\205\377\017\204:\001" [10:00:19.500] record = [10:00:19.500] xlogreader = 0x55aec22b5ee0 [10:00:19.500] private = [10:00:19.500] __func__ = "ReadRecord" [10:00:19.500] #19 0x000055aec07fce39 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1829 [10:00:19.500] xtime = [10:00:19.500] ru0 = {tv = {tv_sec = 1709891786, tv_usec = 689253}, ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 1261}, {ru_maxrss = 16000, __ru_maxrss_word = 16000}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 400, __ru_minflt_word = 400}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 2, __ru_nvcsw_word = 2}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [10:00:19.500] record = 0x7f15434c4690 [10:00:19.500] reachedRecoveryTarget = false [10:00:19.500] replayTLI = 1 [10:00:19.500] __func__ = "PerformWalRecovery" [10:00:19.520] #20 0x000055aec07e6b1c in StartupXLOG () at ../src/backend/access/transam/xlog.c:5723 [10:00:19.520] Insert = [10:00:19.520] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 739}, nextOid = 24576, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1709891784, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 739} [10:00:19.520] wasShutdown = false [10:00:19.520] didCrash = [10:00:19.520] haveTblspcMap = false [10:00:19.520] haveBackupLabel = false [10:00:19.520] EndOfLog = [10:00:19.520] EndOfLogTLI = [10:00:19.520] newTLI = [10:00:19.520] performedWalRecovery = [10:00:19.520] endOfRecoveryInfo = [10:00:19.520] abortedRecPtr = [10:00:19.520] missingContrecPtr = [10:00:19.520] oldestActiveXID = [10:00:19.520] promoted = false [10:00:19.520] __func__ = "StartupXLOG" [10:00:19.520] #21 0x000055aec0c96c22 in StartupProcessMain () at ../src/backend/postmaster/startup.c:252 [10:00:19.520] No locals. [10:00:19.520] #22 0x000055aec0c870d7 in AuxiliaryProcessMain (auxtype=auxtype@entry=B_STARTUP) at ../src/backend/postmaster/auxprocess.c:91 [10:00:19.520] __func__ = "AuxiliaryProcessMain" [10:00:19.520] #23 0x000055aec0c91eb6 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:5351 [10:00:19.520] pid = [10:00:19.520] __func__ = "StartChildProcess" [10:00:19.520] #24 0x000055aec0c96362 in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x55aec22b4db0) at ../src/backend/postmaster/postmaster.c:1460 [10:00:19.520] opt = [10:00:19.520] status = [10:00:19.520] userDoption = [10:00:19.520] listen_addr_saved = false [10:00:19.520] output_config_variable = [10:00:19.520] __func__ = "PostmasterMain" [10:00:19.520] #25 0x000055aec0ac168a in main (argc=4, argv=0x55aec22b4db0) at ../src/backend/main/main.c:197 [10:00:19.520] do_check_root = [10:00:19.520] [10:00:19.520] Thread 1 (LWP 29247): [10:00:19.520] #0 0x00007f1546271fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.520] No symbol table info available. [10:00:19.520] #1 0x00007f154632313a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.520] No symbol table info available. [10:00:19.520] #2 0x00007f1546334a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [10:00:19.520] No symbol table info available.