[13:02:43.975] src/tools/ci/cores_backtrace.sh linux /tmp/cores [13:02:44.702] dumping /tmp/cores/postgres-3-36828.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [13:02:46.538] [New LWP 36828] [13:02:46.538] [New LWP 35786] [13:02:46.538] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [13:02:47.297] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [13:02:48.228] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [13:02:49.447] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [13:02:51.819] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [13:02:51.819] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [13:02:53.867] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [13:02:55.082] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [13:02:56.657] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [13:02:57.714] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [13:02:58.510] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [13:02:59.737] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [13:03:01.350] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [13:03:02.755] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [13:03:03.665] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [13:03:05.198] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [13:03:06.765] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [13:03:07.235] [Thread debugging using libthread_db enabled] [13:03:07.235] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [13:03:08.460] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [13:03:09.853] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [13:03:10.916] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [13:03:11.533] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [13:03:12.296] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [13:03:14.578] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [13:03:15.576] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [13:03:16.402] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [13:03:17.668] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [13:03:18.622] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [13:03:20.284] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [13:03:21.342] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [13:03:22.139] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [13:03:23.518] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [13:03:24.136] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [13:03:25.671] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [13:03:26.434] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [13:03:27.653] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [13:03:28.449] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [13:03:29.559] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [13:03:30.522] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [13:03:31.793] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [13:03:32.434] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [13:03:33.349] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [13:03:34.132] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [13:03:34.624] Core was generated by `postgres: standby2: startup recovering 000000010000000000000006 '. [13:03:34.624] Program terminated with signal SIGQUIT, Quit. [13:03:34.624] #0 0x00007fed29923fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.652] [Current thread is 1 (LWP 36828)] [13:03:34.768] [13:03:34.768] Thread 2 (Thread 0x7fed272baa40 (LWP 35786)): [13:03:34.768] #0 0x00007fed299e6a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.768] No symbol table info available. [13:03:34.768] #1 0x00007fed299d4e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.768] No symbol table info available. [13:03:34.768] #2 0x00007fed299d5497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.768] No symbol table info available. [13:03:34.768] #3 0x00007fed299d4cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.768] No symbol table info available. [13:03:34.768] #4 0x00007fed29930a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.768] No symbol table info available. [13:03:34.768] #5 0x00005564b38627be in RestoreArchivedFile (path=path@entry=0x7fffd3875d50 "", xlogfname=xlogfname@entry=0x7fffd3875d10 "00000002.history", recovername=recovername@entry=0x5564b434db50 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [13:03:34.768] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\066\000\000\000\000\000\000\000\000\321\314\032\264\000\000\000\062\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\034\243\064\264dU\000\000\b\000\000\000\000\000\000\000\231H4\264dU\000\000\070\\\207\323\377\177\000\000\225H4\264dU\000\000\375\307\032\264dU\000\000\241H4\264dU\000\000\231H4\264dU\000\000\070\\\207\323\377\177\000\000\225H4\264dU\000\000\030\251+'\355\177\000\000\233\321\032\264dU", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000"... [13:03:34.768] xlogRestoreCmd = 0x5564b6c2e638 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/040_standby_failover_slots_sync/data/t_040_standby_failover_slots_sync_publisher_data/archives/00000002.history\" \"pg_wal/RECOVERYHISTORY\"" [13:03:34.768] lastRestartPointFname = '0' , "\000\333\064\264dU\000\000\330W\207\323\377\177\000\000\003\333\064\264dU\000\000\030\251+'\355\177\000\000Y\322\032\264dU", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [13:03:34.768] rc = [13:03:34.768] stat_buf = {st_dev = 140656541149464, st_ino = 209180086873, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 48, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 4673356, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 3891110078048108544, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __glibc_reserved = {8, 93891008439068, 140736742250456}} [13:03:34.768] restartSegNo = [13:03:34.768] restartRedoPtr = 93891006745809 [13:03:34.768] restartTli = 0 [13:03:34.768] __func__ = "RestoreArchivedFile" [13:03:34.768] #6 0x00005564b3838bcc in existsTimeLineHistory (probeTLI=probeTLI@entry=2) at ../src/backend/access/transam/timeline.c:235 [13:03:34.768] path = '\000' , "\001\000\000\000\000\000\000\000\275\026\207\263dU\000\000\000a\207\323\377\177\000\000\a\000\000\000\r\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\200a\207\323\377\177\000\000\060a\207\323\377\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "\004_\207\323\377\177\000\000\370\034@\264dU\000\000Q\250ҴdU\000\000\300\250ҴdU\000\000\240\037\000\000\377\377\002\000\a\035@\264dU\000\000\300\250ҴdU\000\000\002\000\000\000\000\000\000\000\372\341\032\264dU\000\000&_\207\323\377\177\000\000$_\207\323\377\177", '\000' ... [13:03:34.768] histfname = "00000002.history", '\000' [13:03:34.768] fd = [13:03:34.768] __func__ = "existsTimeLineHistory" [13:03:34.768] #7 0x00005564b3838cb7 in findNewestTimeLine (startTLI=startTLI@entry=1) at ../src/backend/access/transam/timeline.c:277 [13:03:34.768] newestTLI = 1 [13:03:34.768] probeTLI = 2 [13:03:34.768] #8 0x00005564b386d789 in rescanLatestTimeLine (replayTLI=replayTLI@entry=1, replayLSN=replayLSN@entry=100683512) at ../src/backend/access/transam/xlogrecovery.c:4113 [13:03:34.768] newExpectedTLEs = [13:03:34.768] found = [13:03:34.768] cell = [13:03:34.768] newtarget = [13:03:34.768] oldtarget = 1 [13:03:34.768] currentTle = 0x0 [13:03:34.768] __func__ = "rescanLatestTimeLine" [13:03:34.768] #9 0x00005564b3872939 in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=100683536, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=100683512, replayTLI=1, replayLSN=100683512, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3674 [13:03:34.768] oldSource = XLOG_FROM_STREAM [13:03:34.768] startWalReceiver = false [13:03:34.768] last_fail_time = 764686814861810 [13:03:34.768] now = [13:03:34.768] streaming_reply_sent = true [13:03:34.768] __func__ = "WaitForWALToBecomeAvailable" [13:03:34.768] #10 0x00005564b387336f in XLogPageRead (xlogreader=0x5564b6bdbf80, targetPagePtr=100679680, reqLen=3856, targetRecPtr=100683512, readBuf=0x5564b6c2ae08 "\024\321\006") at ../src/backend/access/transam/xlogrecovery.c:3351 [13:03:34.768] private = 0x5564b6bdbf60 [13:03:34.768] emode = 15 [13:03:34.768] targetPageOff = 16384 [13:03:34.768] targetSegNo = 6 [13:03:34.768] r = [13:03:34.768] __func__ = "XLogPageRead" [13:03:34.768] #11 0x00005564b387b7ca in ReadPageInternal (state=state@entry=0x5564b6bdbf80, pageptr=pageptr@entry=100679680, reqLen=reqLen@entry=3856) at ../src/backend/access/transam/xlogreader.c:1060 [13:03:34.768] readLen = [13:03:34.768] targetPageOff = 16384 [13:03:34.768] targetSegNo = 6 [13:03:34.768] hdr = [13:03:34.768] #12 0x00005564b387d425 in XLogDecodeNextRecord (state=state@entry=0x5564b6bdbf80, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [13:03:34.768] RecPtr = 100683512 [13:03:34.768] record = [13:03:34.768] targetPagePtr = 100679680 [13:03:34.768] randAccess = false [13:03:34.768] len = [13:03:34.768] total_len = [13:03:34.768] targetRecOff = 3832 [13:03:34.768] pageHeaderSize = [13:03:34.768] assembled = false [13:03:34.768] gotheader = [13:03:34.768] readOff = [13:03:34.768] decoded = 0x0 [13:03:34.768] errormsg = 0x5564b43f0101 "LOG" [13:03:34.768] #13 0x00005564b387dfc7 in XLogReadAhead (state=0x5564b6bdbf80, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [13:03:34.768] result = [13:03:34.768] #14 0x00005564b3868909 in XLogPrefetcherNextBlock (pgsr_private=93891050982832, lsn=0x5564b6c40788) at ../src/backend/access/transam/xlogprefetcher.c:492 [13:03:34.768] nonblocking = [13:03:34.768] record = [13:03:34.768] prefetcher = 0x5564b6bdcdb0 [13:03:34.768] reader = 0x5564b6bdbf80 [13:03:34.768] replaying_lsn = 100683464 [13:03:34.768] __func__ = "XLogPrefetcherNextBlock" [13:03:34.768] #15 0x00005564b386ccd2 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [13:03:34.768] No locals. [13:03:34.768] #16 lrq_complete_lsn (lsn=0, lrq=0x5564b6c406b8) at ../src/backend/access/transam/xlogprefetcher.c:290 [13:03:34.768] No locals. [13:03:34.768] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x5564b6bdcdb0, errmsg=errmsg@entry=0x7fffd387a498) at ../src/backend/access/transam/xlogprefetcher.c:1037 [13:03:34.768] record = [13:03:34.768] replayed_up_to = 0 [13:03:34.768] #18 0x00005564b3873b9c in ReadRecord (xlogprefetcher=0x5564b6bdcdb0, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=1) at ../src/backend/access/transam/xlogrecovery.c:3151 [13:03:34.768] errormsg = 0x5564b6bdc790 "" [13:03:34.768] record = [13:03:34.768] xlogreader = 0x5564b6bdbf80 [13:03:34.768] private = [13:03:34.768] __func__ = "ReadRecord" [13:03:34.768] #19 0x00005564b3876d33 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1832 [13:03:34.768] xtime = [13:03:34.768] ru0 = {tv = {tv_sec = 1711371609, tv_usec = 862166}, ru = {ru_utime = {tv_sec = 0, tv_usec = 1561}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 15940, __ru_maxrss_word = 15940}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 404, __ru_minflt_word = 404}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 4, __ru_nvcsw_word = 4}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [13:03:34.768] record = 0x7fed26b74f70 [13:03:34.768] reachedRecoveryTarget = false [13:03:34.787] replayTLI = 1 [13:03:34.787] __func__ = "PerformWalRecovery" [13:03:34.787] #20 0x00005564b38609e0 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5723 [13:03:34.787] Insert = [13:03:34.787] checkPoint = {redo = 83886120, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 749}, nextOid = 24576, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1711371608, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 749} [13:03:34.787] wasShutdown = false [13:03:34.787] didCrash = [13:03:34.787] haveTblspcMap = false [13:03:34.787] haveBackupLabel = true [13:03:34.787] EndOfLog = [13:03:34.787] EndOfLogTLI = [13:03:34.787] newTLI = [13:03:34.787] performedWalRecovery = [13:03:34.787] endOfRecoveryInfo = [13:03:34.787] abortedRecPtr = [13:03:34.787] missingContrecPtr = [13:03:34.787] oldestActiveXID = [13:03:34.787] promoted = false [13:03:34.787] __func__ = "StartupXLOG" [13:03:34.787] #21 0x00005564b3d18311 in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [13:03:34.787] No locals. [13:03:34.787] #22 0x00005564b3d0fadb in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:265 [13:03:34.787] pid = [13:03:34.787] #23 0x00005564b3d119a6 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [13:03:34.787] pid = [13:03:34.787] __func__ = "StartChildProcess" [13:03:34.787] #24 0x00005564b3d17a70 in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x5564b6bdae40) at ../src/backend/postmaster/postmaster.c:1364 [13:03:34.787] opt = [13:03:34.787] status = [13:03:34.787] userDoption = [13:03:34.787] listen_addr_saved = false [13:03:34.787] output_config_variable = [13:03:34.787] __func__ = "PostmasterMain" [13:03:34.787] #25 0x00005564b3b40619 in main (argc=4, argv=0x5564b6bdae40) at ../src/backend/main/main.c:197 [13:03:34.787] do_check_root = [13:03:34.787] [13:03:34.787] Thread 1 (LWP 36828): [13:03:34.787] #0 0x00007fed29923fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.787] No symbol table info available. [13:03:34.787] #1 0x00007fed299d513a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.787] No symbol table info available. [13:03:34.787] #2 0x00007fed299e6a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [13:03:34.787] No symbol table info available.